Introduction
Large-scale pre-trained Video-Language Models (VLMs), such as InternVideo2, have demonstrated exceptional capabilities in general video understanding. However, adapting these massive models for spatio-temporal action detection presents two primary challenges. First, there is a granularity mismatch: VLMs are typically pre-trained on global scene-level descriptions, while action detection requires fine-grained, person-centric understanding. Second, the few-shot overfitting trap makes fine-tuning models with billions of parameters on small datasets (e.g., 15 examples per class) prone to memorization rather than generalization.
To bridge this gap, researchers at NEC Labs America have developed a framework called Group Relative Augmentation, which combines parameter-efficient tuning with novel internal feature manipulation.
Methodology
The proposed framework consists of three main pillars designed to enhance data efficiency and model adaptation:
- Targeted LoRA: The authors utilize Low-Rank Adaptation (LoRA) to inject a small number of trainable parameters into the query and value matrices of the VLM’s transformer blocks. By applying this only to the latter half of the encoder (from layer 30 onwards), the adaptation focuses on high-level visual features while maintaining the general representations captured in early layers.
- Internal FiLM Augmentation: Rather than relying solely on input-level transformations, the method inserts a learnable module after an intermediate layer of the frozen encoder. This module uses Feature-wise Linear Modulation (FiLM) to apply $N$ different "shifts and scales" to internal feature maps. This allows the model to explore task-relevant variations in feature space that frozen layers might otherwise ignore.
- Group-Weighted Loss: To manage the quality of synthetic augmentations, a dynamic weighting strategy is employed. The model calculates the divergence of each augmented version's prediction from the group average. A Gaussian function assigns higher weights to "reasonable" augmentations and down-weights outliers that may represent noise.
Experimental Results
The method was evaluated on two challenging benchmarks for multi-label, multi-person action detection: AVA v2.2 and MOMA 1.0, using a data-efficient setting of approximately 15 training examples per class.
Performance on MOMA
On the MOMA dataset, the proposed method achieved 24.6% mAP, surpassing both the standard LoRA baseline (23.7%) and the state-of-the-art Motion Coherent Augmentation (MCA) (24.3%). Notably, it achieved this while using only 1.27M trainable parameters, compared to the 3.60M required by the other methods.
Performance on AVA
On the AVA dataset, the method reached 25.56% mAP, a significant improvement over the standard LoRA + input augmentation baseline of 24.30%. It performed comparably to MCA (25.63%) but required approximately 65% fewer trainable parameters.
Key Insights and Conclusion
The results suggest that manipulating features inside the network is more effective for adapting large frozen backbones than transforming input pixels alone. Furthermore, the strategic placement of LoRA parameters in the latter half of the vision encoder allows for high performance with significantly reduced computational overhead.
While the method shows great promise for data-efficient action detection, future work will involve exploring its generalizability to other architectures and downstream tasks like video captioning, as well as addressing the current sensitivity to hyperparameters like the number of learned augmentations.
Summary updated