Skip to content

Fix Valgrind issues

Jani Lainema requested to merge lainema/ECM:fix_valgrind_issues into master

Includes fixes for two issues Valgrind complained in ECM 12.0.

First fix relates to LFNST/NSPT in Inter slices and can be considered more serious. It seems in TrQuant::transformNxN (the variant that pre-calculates multiple forward transforms with one call) we calculate sumAbs over the full TU size. However, in Inter slices this function is also used for LFNST/NSPT and in those cases sumAbs is the partially based on whatever happens to be there in the buffers when the function is called. This may lead to somewhat of a random preselection of transforms for the full RDO.

Proposed fix is putting coefficients outside of the active transform area to zero. In RD sense there may be something better to do as only considering surviving coefficients may make the LFNST/NSPT look very competitive in this pre-selection phase.

Second fix relates to initializing m_amvpSbTmvpAmvrEnabledFlag. There seems to already be a merged fix !638 (merged) initializing this parameter when Slice class is created. However, there may still be a possibility (depending on how the Slice classes are used and reused) that data used earlier could carry over if not setting it also for the case of minPoc > 4.

Merge request reports