- The indexing of cost array was corrected by following shuffling pattern of full-RD candidate list employed by MIP.
- ~0.1% coding gain inc. along with ~4% encoding time dec. in AI

This reverts merge request !532

- The indexing of cost array was corrected by following shuffling pattern of full-RD candidate list employed by MIP.
- ~0.1% coding gain inc. along with ~4% encoding time dec. in AI

Moonmo Koo authored 5 years ago and Frank Bossen committed 5 years ago

- Previous name: Reduced Secondary Transform (RST)
- The parameter, LFNST, should be added in cfg file for enabing LFNST, i.e. LFNST = 1

Changed condition for "fast" RA in reduceHadCandList() and renamed variables using camelCase notation

Cosmetic change: Fixed a typo in the name of a variable: "ispIsCurrentWinnder" was replaced with "ispIsCurrentWinner".

keep max TU in #define for possibility of keeping the parameter

TU max/min size are constants in the VVC draft. Reflecting this in the software.

Speed-Up for ISP when JVET_M0464_UNI_MTS is enabled. It can be enabled/disabled with the config. file parameter ISPFast.

When enabled,
-it merges all full RD intra mode lists into one,
-it tests fewer non-DCT-II transforms if ISP is likely to become the best mode and
-it stops testing intra modes for an ISP split if all sub-partitions obtained a zero-cbf.

Results in CTC:

AI -> 0.04% loss, EncT = 97%, DecT = 100%
RA -> 0.01% loss, EncT = 99%, DecT = 100%

Clean-up of 2 parts in the ISP code that are redundant. The changes do not alter the performance in any way, i.e., the results are exactly the same.

1) In the full RD intra mode tests loop (estIntraPredLumaQT) there is the following condition:
    if ( uiOrgMode <= DC_IDX ) -> do X; else-> do X.
Since X is the same in both cases, the redundant code has been removed.

2) In xRecurIntraCodingLumaQT there is the threshold definition

double threshold = nSubPartitions == 2 ? 0.95 : subTuCounter == 0 ? 0.67 : subTuCounter == 1 ? 0.83 : 0.91;

Since subTuCounter can never be 0 (it is at least 1) at the time this expression is evaluated, the code has been simplified in the following way:

double threshold = nSubPartitions == 2 ? 0.95 : subTuCounter == 1 ? 0.83 : 0.91;