Debug bitstream functionality broken

In pre-ECM-11.0 the functionality of DebugBitstream (DecodeBitstream1) with decoding instead of encoding until POC set by DebugPOC (SwitchPOC) setting does not work properly in RA.

There are several scenarios of an error. Note that DebugPOC is set below by default to -1 meaning that the stream is fully parsed and rewritten without actual encoding.

Bug 1. SOLVED in !488 (merged)

BasketballPass at QP 32

  1. (encoding) Produce 1 GOP of normal bitstream:

EncoderApp.exe -c encoder_randomaccess_ecm.cfg -c classD_randomaccess.cfg -c BasketballPass.cfg --SEIDecodedPictureHash=1 --PrintHexPSNR=1 --FramesToBeEncoded=33 -q 32 -b str_enc.bin -o rec_enc.yuv

  1. (recoding) Try to read the bitstream and encode again only POC 31:

EncoderApp.exe -c encoder_randomaccess_ecm.cfg -c classD_randomaccess.cfg -c BasketballPass.cfg --SEIDecodedPictureHash=1 --PrintHexPSNR=1 --FramesToBeEncoded=33 -q 32 -b str_rec.bin -o rec_rec.yuv --DebugBitstream=str_enc.bin

  1. (decoding) Try to read the generated bitstream:

DecoderApp.exe -b str_rec.bin -o rec_dec.yuv

Decoder error at POC 32, i.e. second coded frame ERROR: In function "InputBitstream::readByteAlignment" in ...\source\Lib\CommonLib\BitStream.cpp:399: Code is not '1'

Reason: the issues take place at recoding (elements are read properly, but not written back properly):

  1. Several elements in Picture header, like gdr_or_irap_pic_flag (read 0, written 1)
  2. APS is not incorporated into Picture header (inserted there at normal encoding, not inserted at recoding, therefore not updated)

Bug 2. NOT OBSERVED

Note: sequence BasketballPass is coded with IntraPeriod 64 in CTC

BasketballPass at QP 22:

  1. Encode (can be manually interrupted after 2 frames):

    EncoderApp.exe -c encoder_randomaccess_ecm.cfg -c classD_randomaccess.cfg -c BasketballPass.cfg --SEIDecodedPictureHash=1 --PrintHexPSNR=1 --FramesToBeEncoded=33 -q 22 -b str_enc.bin -o rec_enc.yuv

  2. Re-code:

    EncoderApp.exe -c encoder_randomaccess_ecm.cfg -c classD_randomaccess.cfg -c BasketballPass.cfg --SEIDecodedPictureHash=1 --PrintHexPSNR=1 --FramesToBeEncoded=33 -q 22 -b str_rec.bin -o rec_rec.yuv --DebugBitstream=str_enc.bin

Re-code error at POC 32, i.e. second coded frame: ERROR: In function "OutputBitstream::write" in ...\source\Lib\CommonLib\BitStream.cpp:113: Unsupported parameters

Reason: Uninitialized value of boolean collocated_from_l0_flag_2nd (205) is attempted to be written as a flag inducing a CHECK() error.

Present since ECM-9.0 (JVET_AC0185_ENHANCED_TEMPORAL_MOTION_DERIVATION) where m_colFromL0Flag2nd was introduced.

Bug 3. SOLVED IN !569 (merged) AND !570 (merged)

SlideShow at QP 37:

  1. Encode 1 frame:

    EncoderApp.exe -c encoder_randomaccess_ecm.cfg -c classF.cfg -c SlideShow.cfg --SEIDecodedPictureHash=1 --PrintHexPSNR=1 --FramesToBeEncoded=33 -q 37 -b str_enc.bin -o rec_enc.yuv

  2. Re-code:

    EncoderApp.exe -c encoder_randomaccess_ecm.cfg -c classF.cfg -c SlideShow.cfg --SEIDecodedPictureHash=1 --PrintHexPSNR=1 --FramesToBeEncoded=33 -q 37 -b str_rec.bin -o rec_rec.yuv --DebugBitstream=str_enc.bin

  3. Decode:

    DecoderApp.exe -b str_rec.bin -o rec_dec.yuv

Decoder error at POC 0, i.e. first coded frame ERROR: In function "DecCu::xDeriveCUMV" in C:\Users\gverba\source\repos\ECM2\source\Lib\DecoderLib\DecCu.cpp:4367: invalid block vector for IBC detected.

Mismatches (ordered) noted by tracing:

  1. (POC=0) cu.curPLT[compID][idx] (CABACReader.cpp:3096 / CABACWriter.cpp:2721) is not coded by encoder, but coded at re-encoding
  2. (POC=32) sps_obmc_flag=0 at encoder | =1 at re-encoding (possibly not rewritten after slice 0 having value 1)
  3. (POC=32) ph_fpel_mbvd_enabled_flag=1 at encoder | =0 at re-encoding (possibly not rewritten after slice 0 having value 0)

Items 1,2 are observed in ECM-8.0, maybe introduced even earlier. Item 3 is introdued in ECM-9.0 together with the fpel flag itself.

Bug 4. SOLVED IN !566 (merged)

RaceHorses at Q27:

  1. Encode 1 frame:

    EncoderApp.exe -c encoder_randomaccess_ecm.cfg -c classD_randomaccess.cfg -c RaceHorses.cfg --SEIDecodedPictureHash=1 --PrintHexPSNR=1 --FramesToBeEncoded=1 -q 27 -b str_enc.bin -o rec_enc.yuv

  2. Re-code:

    EncoderApp.exe -c encoder_randomaccess_ecm.cfg -c classD_randomaccess.cfg -c RaceHorses.cfg --SEIDecodedPictureHash=1 --PrintHexPSNR=1 --FramesToBeEncoded=1 -q 27 -b str_rec.bin -o rec_rec.yuv --DebugBitstream=str_enc.bin

  3. Decode:

    DecoderApp.exe -b str_rec.bin -o rec_dec.yuv

Decoder mismatches starting from POC 0

Tracing shows mismatch in writing ALF APS at POC 0, element slice_cc_alf_cr_enabled_flag (VLCWriter.cpp:2977):Encoder: 1 | DebugBitstream: 0

Observed in ECM-8.0, maybe introduced even earlier.

Edited by Gleb Verba