Fix #1497: Encoder crash for RGB content when LFNST and TransformSkip disabled (!2104) · Merge requests · jvet / VVCSoftware_VTM

Adrian Browne requested to merge adybrowne/VVCSoftware_VTM:FIX_1497_RGB_LFNST_TS into master Jul 28, 2021

In xRecurIntraCodingACTQT, if LFNST and TransformSkip are disabled, 5 modes are added to trModes (numbered 0, 2, 3, 4 and 5) and the value of nNumTransformCands is calculated correctly.

However this value is not used in the calculation of the upperbound of the modeId loop. Instead the value of lastCheckId is used which does not recognise that TS is disabled.

Fix is to use same loop as in xRecurIntraCodingLumaQT.

Fix #1497: Encoder crash for RGB content when LFNST and TransformSkip disabled

Merge request reports