Skip to content
Snippets Groups Projects

Fix #484: handeling the case that all coding modes of a CU do not get coding results

Merged Fix #484: handeling the case that all coding modes of a CU do not get coding results
All threads resolved!
Merged Yin Zhao requested to merge (removed):Branch_VTM-6.0 into master
All threads resolved!

In xCompressCU(), several coding modes (e.g., skip/merge, affine, IMV, IBC, intra, modeSplit) are tried, but in some case all tried modes do not finish encoding because early termination (either from fast algorithm or normative checks) occurs inside encoding of each mode. In this case, both tempCS and bestCS have MAX_DOUBLE cost, and bestCS does not containing cu coding data.

Before the integration of JVET_O0050, any CU is allowed to do intra coding, and the RDO process does not skip intra coding mode (i.e., xCheckRDCostIntra()) if no inter coding mode gets complete coding results. Intra mode is like a backup plan of the RDO to ensure each CU has a valid coding mode. However, in an inter coding region (e.g., a 4x8 CU with modeTYpe == MODE_TYPE_INTER), only inter coding modes are tried, but the inter coding modes do not guarantee at least one valid coding mode.

So, if no valid coding mode after trying all inter coding modes in RDO, one way to avoid encoder crash is to return before saving best coding mode information at the end of xCompressCU(), as implemented by this merge request.

Merge request reports

Merged by avatar (Jan 21, 2025 8:22pm UTC)

Loading

Pipeline #2590 passed

Pipeline passed for 5f250e00 on master

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Xiang Li resolved all threads

    resolved all threads

  • merged

  • Xiang Li mentioned in commit 5f250e00

    mentioned in commit 5f250e00

  • Please register or sign in to reply
    Loading