Skip to content

JVET-AC0116 : neural network-based intra prediction with learned mapping to VVC intra prediction modes.

Thierry Dumas requested to merge jvet-ab-ee1/VVCSoftware_VTM:EE1-3.2 into VTM-11.0_nnvc

Target of the MR

The MR aims at integrating JVET-AC0116, which was accepted during the 29th JVET meeting (January 2023), into VTM-11-NNVC.

Description of the MR

This MR contains the code of the neural network-based intra prediction mode inside VTM-11-NNVC for inference. It also contains the code for training entirely the neural networks involved in the neural network-based intra prediction mode inside VTM-11-NNVC.

  1. Code of the neural network-based intra prediction mode for inference.

    • "CMakeLists.txt" is almost unchanged with respect to its original version. The only change is the addition of a "warning without error" for MSVC at line 101. This "warning without error" is needed for compiling the code of the two Filter-Sets when NN_FIXED_POINT_IMPLEMENTATION is equal to 0.
    • In "source/Lib/CommonLib/TypeDef.h", four macros are added: JVET_AB0149_ACTIVATE_WRITER_BLOCK_STATS, JVET_AB0149_INTRA_PRED, JVET_AB0149_SEARCH_PNN, and JVET_AB0149_TM_INF_PNN. These four variables can hardly be merged because, during the training phase, at three different training iterations, and during the test phase, four different versions of VTM-11-NNVC with the neural network-based intra prediction are needed, and these four versions are generated by changing the values of these four macros. These four macros have the prefix "JVET_AB0149" as EE1-3.2 reuses exactly the code of "JVET-AB0149" in the inference part.

  2. Code of the training of the neural networks involved in the neural network-based intra prediction mode.

    • The entire code for training these neural networks is located in the directory at "training/training_intra_prediction".
    • "training/training_intra_prediction/CMakeLists.txt" enables to compile three different versions of VTM-11-NNVC with the neural network-based intra prediction mode, each version being used at a different training iteration.
    • The outputs of the compilations of different versions of VTM-11-NNVC used by the training are separated from the outputs of the compilation of VTM-11-NNVC used by the final rate-distortion tests. Indeed, the outputs of the compilations of different versions of VTM-11-NNVC used by the training are stored in "training/training_intra_prediction/lib" and "training/training_intra_prediction/bin".
Edited by Thierry Dumas

Merge request reports