COMEDIAN: Self-supervised learning and knowledge distillation for action spotting using transformers

Julien Denize (1) , Mykola Liashuha (1) , Jaonary Rabarisoa (1) , Astrid Orcesi (1) , Romain Hérault (2, 3)

1 DIASI (CEA, LIST) - Département Intelligence Ambiante et Systèmes Interactifs
2 Equipe Image - Laboratoire GREYC - UMR6072
3 App - LITIS - Equipe Apprentissage

We present COMEDIAN, a novel pipeline to initialize spatiotemporal transformers for action spotting, which involves self-supervised learning and knowledge distillation. Action spotting is a timestamp-level temporal action detection task. Our pipeline consists of three steps, with two initialization stages. First, we perform self-supervised initialization of a spatial transformer using short videos as input. Additionally, we initialize a temporal transformer that enhances the spatial transformer's outputs with global context through knowledge distillation from a pre-computed feature bank aligned with each short video segment. In the final step, we fine-tune the transformers to the action spotting task. The experiments, conducted on the SoccerNet-v2 dataset, demonstrate state-of-the-art performance and validate the effectiveness of COMEDIAN's pretraining paradigm. Our results highlight several advantages of our pretraining pipeline, including improved performance and faster convergence compared to non-pretrained models.

Mots clés: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); FOS: Computer and information sciences

Domaines: Machine Learning [stat.ML]

Julien Denize, Mykola Liashuha, Jaonary Rabarisoa, Astrid Orcesi, Romain Hérault. COMEDIAN: Self-supervised learning and knowledge distillation for action spotting using transformers. WACV 2024 - IEEE/CVF Winter Conference on Applications of Computer Vision, Jan 2024, Waikoloa, United States. ⟨10.48550/arXiv.2309.01270⟩. ⟨hal-04462159⟩