Skip to content

#179: fix of reshaper implementation (encoder-side RDcost) when lumaDQP is on for HDR

Taoran Lu requested to merge tlu/VVCSoftware_VTM:partial-fix-reshaper-lumaDQP-for-HDR into master

Current reshaper encoder implementation have wrong condition for WSSE RDCost calculation when lumaDQP is enabled and reshaping is off. This merge request fixed the issue. Also the luma weight table initialization part for lumaDQP is changed to use the reshaper way that supports general bitdepth (therefore not static size 1024). Also, an encoder cfg check is set to prevent the use lumaDQP together with reshaper for HDR.

This is a partial fix on reshaper side for HDR Anchor simulation. There are still issues observed for lumaDQP interaction with (possibly) JVET_M0102_INTRA_SUBPARTITIONS that crashes encoding for HDR CTC setting (lumaDQP=1, reshape=0). The issue is observed no matter JVET_M0427_INLOOP_RESHAPER is set to 0 or 1 in typeDef.h and WCG_EXT=1 for HDR Anchor.

Edited by Frank Bossen

Merge request reports