Skip to content

Fix #484: handeling the case that all coding modes of a CU do not get coding results

Yin Zhao requested to merge (removed):Branch_VTM-6.0 into master

In xCompressCU(), several coding modes (e.g., skip/merge, affine, IMV, IBC, intra, modeSplit) are tried, but in some case all tried modes do not finish encoding because early termination (either from fast algorithm or normative checks) occurs inside encoding of each mode. In this case, both tempCS and bestCS have MAX_DOUBLE cost, and bestCS does not containing cu coding data.

Before the integration of JVET_O0050, any CU is allowed to do intra coding, and the RDO process does not skip intra coding mode (i.e., xCheckRDCostIntra()) if no inter coding mode gets complete coding results. Intra mode is like a backup plan of the RDO to ensure each CU has a valid coding mode. However, in an inter coding region (e.g., a 4x8 CU with modeTYpe == MODE_TYPE_INTER), only inter coding modes are tried, but the inter coding modes do not guarantee at least one valid coding mode.

So, if no valid coding mode after trying all inter coding modes in RDO, one way to avoid encoder crash is to return before saving best coding mode information at the end of xCompressCU(), as implemented by this merge request.

Merge request reports