m_modeCostStore in estIntraPredLumaQT() is used without being initialized in ECM-4.0
m_modeCostStore is used without being initialized (or being exploited with the residue of the previous cost) by the encoding optimization code in line 1437, in IntraSearch.cpp. This code skips storing the cost of the current mode and it affects the candidate selection for the full RD when an LFNST index is larger than 0. The proposed fixes are attached.
- The first option is to initialize the cost as the max value before continue.
- The second option is just remove this optimization code.
Since DIMD mode with only mode is rare, I believe it does not affect the encoding time significantly.
Edited by Sunmi Yoo