- Fix bugs in N0280 - Transform skip residual coding.

Moonmo Koo authored 5 years ago and Frank Bossen committed 5 years ago

- Previous name: Reduced Secondary Transform (RST)
- The parameter, LFNST, should be added in cfg file for enabing LFNST, i.e. LFNST = 1

Merge branch 'N0280-ResidualCodingTransformSkip' of https://vcgit.hhi.fraunhofer.de/nguyen/VVCSoftware_VTM into N0280-ResidualCodingTransformSkip

- Harmonize coefficient group sizes of chroma TBs to those used in luma TBs
- For 2x4/4x2 chroma TBs, retain 2x2 CG size.

TCoeff is not always an int. Using the TCoeff type as template parameter
is preferred.

- The M0297 zero-out is also supported for SBT now.
- The zero-out scanning code is moved out of residual_coding_subblock.

Use a single array with elements that comprise scan x position, y
position, and index into block instead of separate arrays.

Simplify array initialization code

The integration of JVET_M0297_32PT_MTS_ZERO_OUT adds unnecessary code as
well as unnecessary processing in the encoder quantization process. -
The suggested change simplifies the code and avoids unnecessary
processing for the zero-out region.

The suggested changes fix the following two issues:

[1]
For blocks with width>32 or height>32, the scan created in initRom()
(file Rom.cpp) are only valid for the non-zero-out region. However, in
Rom::xInitScanArrays() (file DepQuant.cpp), the zero-out region is also
used. This has the effect that the array raster2id[] contains wrong
values. As a consequence, wrong neighboring positions cpos[] are derived.

Even though a previous fix (fix #161) ensures that the positions are
clipped to a value inside the subblock, these positions could still be
wrong. It is suggested to fix the issue by correctly considering the
zero-out regions for larger blocks (which is the actual source of the problem).

[2]
In State::updateStateEOS(..), the array m_absLevelAndCtxInit should be
set equal to zero if a skipped subblock is selected. Otherwise, it could
be possible lead to a usage of wrongs contexts for following transform
coefficients.

In Rom::xInitScanArrays()

Make sure that the position of the sample in the template falls
in the same subblock as the current sample before subtracting the
offset of the begining of the subblock.

Refactor code to make lines fit within a reasonable of characters.

Large blocks, i.e. with either width or height equal to 64, have
fewer coefficients. The number of coefficients should thus be
set accordingly

There is a conflict with JVET_M0427_INLOOP_RESHAPER that is triggering a debug assertion that checks that the width is not 1.

Fixed transition point between GR and TU + EG(k) escape codes for coefficient coding. Use limited prefix length for escape codes.

Move the gtr2 flag (the abs_level_gt3_flag in text) after the par_level_flag in the first coding pass