JVET-AC0055: content-adaptive post-filter training scripts

JVET-AC0055: content-adaptive post-filter. Training scripts.

marked this merge request as draft

added 1 commit

• 3bcd3dd6 - JVET-AC0055: content-adaptive post-filter

Compare with previous version

marked this merge request as ready

changed title from JVET-AC0055: content-adaptive post-filter to JVET-AC0055: content-adaptive post-filter training scripts

Upon the merging of the inference code, there is now a conflict in the .gitattributes file. I am unable to resolve this since this branch is private. Please just accept both changes. I can then merge the request

added 3 commits

• 3bcd3dd6...87c147b2 - 2 commits from branch jvet-ahg-nnvc:VTM-11.0_nnvc
• 6bc33eca - Rebase to VTM-11.0_nnvc

Compare with previous version

added 1 commit

• bb153300 - Update readme

Compare with previous version

this MR contains a lot of files: patch file, saved model, pb files... are they all necessary for the training?

The patch is for the NNR/NNC weight-update computation. It is to be applied on top of NCTM repository.

The TensorFlow models were used as initialisation. They are available in JVET-W-EE1 repository too. Should they be removed?

You mean they are use to do "pseudo-training" stage? If so, we can keep them for people wanting to do only "pseudo-training". Thanks.

It is a kind of "pseudo-training":

1. The base models (before over-fitting) are initialised using the TensorFlow models in the MR
2. Those base models further fine-tuned
3. The base models are over-fitted.

Just to be sure: the models in the MR are the one trained on BVI and DVI2K. They are the ones used to overfit on the each particular sequence?

No, the models in the MR are the models from step (1).

The training scripts generate the models from steps (2) and (3).

So on what were trained the models in step 1?

Training scripts allow to fine tuned on BVI/DIV2K to give step 2. Then models from step 2 are overfitted for each sequence.

The models in step (1) are from JVET-W0131, they were trained with BVI-DVC. They were used to initialise the models in (2) which are further fine-tuned on BVI-DVC and DIV2K.

The models in step (1) were trained by the proponents of JVET-W0131.

I just want to confirm that the MR contains everything to train the models from scratch, not just fine-tuning models from non accessible origin.

I understand. In that case, the scripts need to be modified to support training from scratch.

Should both the pseudo-training and training from scratch be supported; or only the training from scratch?

Yes, please modify the scripts to allow training from scratch. Was there a training xcheck of JVET-W0131?

From my understanding, the MR contains everything to fully replicate the proponent's training process from scratch. The difference here compared to the ILF proposals, is that the proponent never trained the W0131 ILF models from scratch, but rather started with the pre-trained W0131 ILF models, and then converted these to post filter models using the training in step 2 (and subsequently to sequence/QP-dependant overfitted models in step 3). I would consider a training from scratch for this proposal to start with the pretrained ILF models from W0131, as was done originally by the proponents. If training from scratch must also include the training of the W0131 ILF models (i.e from a random weight initialization), something not done by proponents, then it is possible to get markedly different results. Is my understanding correct, Maria? What are your thoughts on this training-from-scratch distinction, Maria and Franck?

Training cross-check was done by Ericsson in JVET-AC0331. They started with the pre-trained W0131 ILF models

" I would consider a training from scratch for this proposal to start with the pretrained ILF models from W0131" : no, it would not be! The issue, well discussed during meetings, with not having training scripts/xcheck from scratch is that it makes modification/improvement of the model impossible (if the training cannot be redo). The issue needs to raised to the group. My understanding is that retraining from scratch should be ok with a small modification of the current training script, so we just need to check that similar results can be obtained.

I understand, but I think there are two parts here: JVET-W0131 and JVET-AC0055. My understanding of training from scratch is that the proponent's complete training process is replicated. I believe this MR handles this for JVET-AC0055 (i.e one can replicate everything proponent did). I am not sure if JVET-W0131 was training cross-checked. In any case, the software for this is not currently included in the common software base, and so it would be good to also have this. If W0131 was not training cross-checked, then yes, this brings the completeness of AC0055's training cross-check into discussion.

Yes, I think we can accept the MR because it brings what was discussed during the meeting. The concern about the origin/validity of the W0131 model should be bring to discussion. Hopefully, a training from scratch using the same scripts would give the same results. @msantamaria did you try to retrain from scratch?

No, for this proposal, the base models were not trained from scratch.

In addition, as @sameadie mentioned, the MR includes everything to replicate what we did.

Yes, the MR is aligned with what was accepted. The remaining question is are we able to get the models from W0131 from scratch. My understanding is that it is still an open question.

@sameadie can you accept the MR if it is OK with you?

marked this merge request as draft

marked this merge request as ready

assigned to @sameadie

added 1 commit

• 751a31ab - updating .gitattributes

Compare with previous version

added 5 commits

• bb153300...120c5810 - 4 commits from branch jvet-ahg-nnvc:VTM-11.0_nnvc
• ee1b1658 - Rebase to VTM-11.0_nnvc

Compare with previous version

mentioned in commit f170efb7

merged