JVET-AC0116 : neural network-based intra prediction with learned mapping to VVC intra prediction modes.

Target of the MR

The MR aims at integrating JVET-AC0116, which was accepted during the 29th JVET meeting (January 2023), into VTM-11-NNVC.

Description of the MR

This MR contains the code of the neural network-based intra prediction mode inside VTM-11-NNVC for inference. It also contains the code for training entirely the neural networks involved in the neural network-based intra prediction mode inside VTM-11-NNVC.

1. Code of the neural network-based intra prediction mode for inference.

   • "CMakeLists.txt" is almost unchanged with respect to its original version. The only change is the addition of a "warning without error" for MSVC at line 101. This "warning without error" is needed for compiling the code of the two Filter-Sets when NN_FIXED_POINT_IMPLEMENTATION is equal to 0.
   • In "source/Lib/CommonLib/TypeDef.h", four macros are added: JVET_AB0149_ACTIVATE_WRITER_BLOCK_STATS, JVET_AB0149_INTRA_PRED, JVET_AB0149_SEARCH_PNN, and JVET_AB0149_TM_INF_PNN. These four variables can hardly be merged because, during the training phase, at three different training iterations, and during the test phase, four different versions of VTM-11-NNVC with the neural network-based intra prediction are needed, and these four versions are generated by changing the values of these four macros. These four macros have the prefix "JVET_AB0149" as EE1-3.2 reuses exactly the code of "JVET-AB0149" in the inference part.

2. Code of the training of the neural networks involved in the neural network-based intra prediction mode.

   • The entire code for training these neural networks is located in the directory at "training/training_intra_prediction".
   • "training/training_intra_prediction/CMakeLists.txt" enables to compile three different versions of VTM-11-NNVC with the neural network-based intra prediction mode, each version being used at a different training iteration.
   • The outputs of the compilations of different versions of VTM-11-NNVC used by the training are separated from the outputs of the compilation of VTM-11-NNVC used by the final rate-distortion tests. Indeed, the outputs of the compilations of different versions of VTM-11-NNVC used by the training are stored in "training/training_intra_prediction/lib" and "training/training_intra_prediction/bin".