Out-of-Bounds Access in xIntraCodingTUBlock with JVET_AH0209_PDP
Possibly related to #69. !676 (merged) does not resolve the issue for me.
Here is an Address Sanitizer report:
/nfs/frankp/ecm/source/Lib/EncoderLib/IntraSearch.cpp:9634:43: runtime error: index 98 out of bounds for type 'int [67]'
/nfs/frankp/ecm/source/Lib/EncoderLib/IntraSearch.cpp:9634:12: runtime error: load of address 0x563ac82b31a8 with insufficient space for an object of type 'const int'
0x563ac82b31a8: note: pointer points here
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 32 34 20 34
^
=================================================================
==32647==ERROR: AddressSanitizer: global-buffer-overflow on address 0x563ac82b31a8 at pc 0x563ac679f053 bp 0x7ffe22dc7120 sp 0x7ffe22dc7110
READ of size 4 at 0x563ac82b31a8 thread T0
#0 0x563ac679f052 in IntraSearch::xIntraCodingTUBlock(TransformUnit&, ComponentID const&, unsigned long&, int const&, unsigned int*, std::vector<std::pair<int, bool>, std::allocator<std::pair<int, bool> > >*, bool, InterPrediction*) /nfs/frankp/ecm/source/Lib/EncoderLib/IntraSearch.cpp:9634
#1 0x563ac67d8061 in IntraSearch::xRecurIntraCodingLumaQT(CodingStructure&, Partitioner&, double, int, PartSplit, bool, bool, int, int, bool, InterPrediction*) /nfs/frankp/ecm/source/Lib/EncoderLib/IntraSearch.cpp:11078
#2 0x563ac66c6c8e in IntraSearch::estIntraPredLumaQT(CodingUnit&, Partitioner&, double, bool, int, int, bool, CodingStructure*, InterPrediction*) /nfs/frankp/ecm/source/Lib/EncoderLib/IntraSearch.cpp:3546
#3 0x563ac6f0dd31 in EncCu::xCheckRDCostIntra(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&, bool) /nfs/frankp/ecm/source/Lib/EncoderLib/EncCu.cpp:3979
#4 0x563ac6eb58d0 in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /nfs/frankp/ecm/source/Lib/EncoderLib/EncCu.cpp:1694
#5 0x563ac6ed8200 in EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&, double*) /nfs/frankp/ecm/source/Lib/EncoderLib/EncCu.cpp:2674
#6 0x563ac6ebad19 in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /nfs/frankp/ecm/source/Lib/EncoderLib/EncCu.cpp:1870
#7 0x563ac6ed8200 in EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&, double*) /nfs/frankp/ecm/source/Lib/EncoderLib/EncCu.cpp:2674
#8 0x563ac6ebad19 in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /nfs/frankp/ecm/source/Lib/EncoderLib/EncCu.cpp:1870
#9 0x563ac6ed8200 in EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&, double*) /nfs/frankp/ecm/source/Lib/EncoderLib/EncCu.cpp:2674
#10 0x563ac6ebad19 in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /nfs/frankp/ecm/source/Lib/EncoderLib/EncCu.cpp:1870
#11 0x563ac6ed8200 in EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&, double*) /nfs/frankp/ecm/source/Lib/EncoderLib/EncCu.cpp:2674
#12 0x563ac6ebad19 in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /nfs/frankp/ecm/source/Lib/EncoderLib/EncCu.cpp:1870
#13 0x563ac6ed8200 in EncCu::xCheckModeSplit(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&, double*) /nfs/frankp/ecm/source/Lib/EncoderLib/EncCu.cpp:2674
#14 0x563ac6ebad19 in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /nfs/frankp/ecm/source/Lib/EncoderLib/EncCu.cpp:1870
#15 0x563ac6ee6cf7 in EncCu::xCheckRDCostSeparateTreeIntra(CodingStructure*&, CodingStructure*&, Partitioner&, EncTestMode const&) /nfs/frankp/ecm/source/Lib/EncoderLib/EncCu.cpp:3091
#16 0x563ac6eb722b in EncCu::xCompressCU(CodingStructure*&, CodingStructure*&, Partitioner&, double) /nfs/frankp/ecm/source/Lib/EncoderLib/EncCu.cpp:1723
#17 0x563ac6e862a6 in EncCu::compressCtu(CodingStructure&, UnitArea const&, unsigned int, int const*, int const*) /nfs/frankp/ecm/source/Lib/EncoderLib/EncCu.cpp:712
#18 0x563ac629c0bf in EncSlice::encodeCtus(Picture*, bool, bool, EncLib*) /nfs/frankp/ecm/source/Lib/EncoderLib/EncSlice.cpp:2307
#19 0x563ac6285df1 in EncSlice::compressSlice(Picture*, bool, bool) /nfs/frankp/ecm/source/Lib/EncoderLib/EncSlice.cpp:1805
#20 0x563ac5e9e73e in EncGOP::compressGOP(int, int, std::__cxx11::list<Picture*, std::allocator<Picture*> >&, std::__cxx11::list<UnitBuf<short>*, std::allocator<UnitBuf<short>*> >&, bool, bool, InputColourSpaceConversion, bool, bool, bool, int) /nfs/frankp/ecm/source/Lib/EncoderLib/EncGOP.cpp:3631
#21 0x563ac600f83b in EncLib::encode(InputColourSpaceConversion, std::__cxx11::list<UnitBuf<short>*, std::allocator<UnitBuf<short>*> >&, int&) /nfs/frankp/ecm/source/Lib/EncoderLib/EncLib.cpp:1220
#22 0x563ac35ecdba in EncApp::encode() /nfs/frankp/ecm/source/App/EncoderApp/EncApp.cpp:1919
#23 0x563ac391bc5f in main /nfs/frankp/ecm/source/App/EncoderApp/encmain.cpp:281
#24 0x7fb1d2f07209 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#25 0x7fb1d2f072bb in __libc_start_main_impl ../csu/libc-start.c:389
#26 0x563ac35130c0 in _start (/nfs/frankp/ecm/bin/EncoderAppStaticd+0x755d0c0)
0x563ac82b31a8 is located 27 bytes to the right of global variable '*.LC17' defined in '/nfs/frankp/ecm/source/Lib/CommonLib/Rom.cpp' (0x563ac82b3160) of size 45
'*.LC17' is ascii string '/nfs/frankp/ecm/source/Lib/CommonLib/Rom.cpp'
SUMMARY: AddressSanitizer: global-buffer-overflow /nfs/frankp/ecm/source/Lib/EncoderLib/IntraSearch.cpp:9634 in IntraSearch::xIntraCodingTUBlock(TransformUnit&, ComponentID const&, unsigned long&, int const&, unsigned int*, std::vector<std::pair<int, bool>, std::allocator<std::pair<int, bool> > >*, bool, InterPrediction*)
Shadow bytes around the buggy address:
0x0ac7d904e5e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0ac7d904e5f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 04 f9 f9
0x0ac7d904e600: f9 f9 f9 f9 00 00 00 00 00 00 00 00 00 00 00 00
0x0ac7d904e610: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0ac7d904e620: 00 00 00 00 00 04 f9 f9 f9 f9 f9 f9 00 00 00 00
=>0x0ac7d904e630: 00 05 f9 f9 f9[f9]f9 f9 00 00 00 00 00 00 00 00
0x0ac7d904e640: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0ac7d904e650: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0ac7d904e660: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0ac7d904e670: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0ac7d904e680: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==32647==ABORTINGSteps to reproduce:
$ git clone https://vcgit.hhi.fraunhofer.de/ecm/ECM.git && cd ECM
$ cmake -E make_directory build
$ cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS="-fsanitize=address" -DCMAKE_CXX_FLAGS="-fsanitize=address"
$ cmake --build build --parallel
$ ./bin/EncoderAppStaticd -c cfg/encoder_intra_ecm.cfg -c cfg/per-sequence/BasketballPass.cfg -i <PATH TO D_BasketballPass_416x240_50Hz_8bit_P420.yuv> -ip 1 -fs 0 -f 1 -q 37 -o /dev/null