dev_notes_hop.md

Schedule

1- extract VTM data for intra training. (1 week)

• can be started now
• extract json, YUV, md5 etc.

2- train intra model (2 weeks)

• need a first working training script

3- validate intra model

• need a first working NNVC-5.0 VTM (only for intra frame)

Software structure

Main classes

• NNFilterHOP

class NNFilterHOP {
  init: call once
  filter: call to filter the whole picture
  filterBlock: call per block. storage own by the class NNFilterHOP. No scaling done, output on 14 bits (log2OutputScale)
  scaleResidualBlock: perform residual scaling
  resizeInputs: resize the inputs to target shape (take size of extended block as input)
};

• EncNNFilterHOP

class EncNNFilterHOP {
  chooseParameters: perform the RDO (by calling parameterSearch) to get the best parameters, e.g. qp offsets, scaling factors
  parameterSearch: search to find out the best QP offsets
  scaleFactorDerivation: derive residual scaling factors with least square methods
  scalePicture: call at encoder to scale the whole picture with the above derived scaling factors
  setNnlfHopInferGranularity: call at encoder to adaptively determine the inference size
};

• encoder algorithm

compressGOP() {
 ...
#if NN_HOP_UNIFIED
      if (pcSlice->getSPS()->getNnlfHopEnabledFlag())
      {
        m_nnfilterHOP.initCabac( m_pcEncLib->getCABACEncoder(), m_pcEncLib->getCtxCache(), *pcSlice);
        m_nnfilterHOP.setPicprms(&pcPic->m_picprm);
        m_nnfilterHOP.chooseParameters(*pcPic);
        pcSlice->setNnlfHopParameters(m_nnfilterHOP.getSliceprms());
      }
#endif

• NNFilterHOP::FilterParameters

// parameters needed to filter the picture
struct FilterParameters
{
  int              block_size;
  int              extension;
  int              nb_blocks_width;
  int              nb_blocks_height;
  int              prmNum;
  std::vector<int> prmId; // -1: off
  SliceParameters  sprm;
};

• NNFilterHOP::SliceParameters

// parameters signaled at slice level
struct SliceParameters {
  int mode; // -1: off
  int scaleFlag;
  int scale[MAX_NUM_COMPONENT][2];
  
  SliceParameters()
  {
    reset();
  }
  void reset()
  {
    mode = -1;
    scaleFlag = -1;
    std::memset( scale, 0, sizeof( scale ) );
  }
};

• CABAC write/readNnlfHopParameters: signal parameters of each block
• VLC wirte/readHopScalingMode: signal scaling mode of each slice static const CtxSet nnlfHopParams; prepare for signaling contexts.

Algorithms

• rdo: QP adaptation, residual scaling (separate scaling factors to be used for Cb and Cr)
• signaling: data in FilterParameters -block_size: signalled in SPS -extension: signalled in SPS -prmNum: signalled in SPS -prmId: signalled in coding_tree_unit -sprm.mode: signalled at slice header -sprm.scaleFlag: signalled at slice header -sprm.scale: signalled at slice header

Model

• model inputs: see NNFilterHOP::filterBlock to see model inputs and inputs quantizers.
• assume input in SADL as quantizer >=10 (because perform a left shift)
• assume output in SADL as quantizer <=14 (because perform a right shift)
• model output: the 6 data plan (4Y and UV) are unshuffled in C++ side (see NNFilterHOP.cpp:extractOutputs)

Other changes

• pass to C++-17 (aligned with VTM, simplify generic code for int16/float handling)
• added some const in prepareInputs