Group Relative Augmentation for Data Efficient Action Detection

Deep Anil Patel, Iain Melvin, Zachary Izzo, Martin Renqiang MinarXiv 2025 · DOI · arXiv

Introduction

Large-scale pre-trained Video-Language Models (VLMs), such as InternVideo2, have demonstrated exceptional capabilities in general video understanding. However, adapting these massive models for spatio-temporal action detection presents two primary challenges. First, there is a granularity mismatch: VLMs are typically pre-trained on global scene-level descriptions, while action detection requires fine-grained, person-centric understanding. Second, the few-shot overfitting trap makes fine-tuning models with billions of parameters on small datasets (e.g., 15 examples per class) prone to memorization rather than generalization.

To bridge this gap, researchers at NEC Labs America have developed a framework called Group Relative Augmentation, which combines parameter-efficient tuning with novel internal feature manipulation.

Methodology

The proposed framework consists of three main pillars designed to enhance data efficiency and model adaptation:

  • Targeted LoRA: The authors utilize Low-Rank Adaptation (LoRA) to inject a small number of trainable parameters into the query and value matrices of the VLM’s transformer blocks. By applying this only to the latter half of the encoder (from layer 30 onwards), the adaptation focuses on high-level visual features while maintaining the general representations captured in early layers.
  • Internal FiLM Augmentation: Rather than relying solely on input-level transformations, the method inserts a learnable module after an intermediate layer of the frozen encoder. This module uses Feature-wise Linear Modulation (FiLM) to apply $N$ different "shifts and scales" to internal feature maps. This allows the model to explore task-relevant variations in feature space that frozen layers might otherwise ignore.
  • Group-Weighted Loss: To manage the quality of synthetic augmentations, a dynamic weighting strategy is employed. The model calculates the divergence of each augmented version's prediction from the group average. A Gaussian function assigns higher weights to "reasonable" augmentations and down-weights outliers that may represent noise.

Experimental Results

The method was evaluated on two challenging benchmarks for multi-label, multi-person action detection: AVA v2.2 and MOMA 1.0, using a data-efficient setting of approximately 15 training examples per class.

Performance on MOMA

On the MOMA dataset, the proposed method achieved 24.6% mAP, surpassing both the standard LoRA baseline (23.7%) and the state-of-the-art Motion Coherent Augmentation (MCA) (24.3%). Notably, it achieved this while using only 1.27M trainable parameters, compared to the 3.60M required by the other methods.

Performance comparison on the MOMA 1.0 dataset.

Performance on AVA

On the AVA dataset, the method reached 25.56% mAP, a significant improvement over the standard LoRA + input augmentation baseline of 24.30%. It performed comparably to MCA (25.63%) but required approximately 65% fewer trainable parameters.

Performance comparison on the AVA v2.2 dataset.

Key Insights and Conclusion

The results suggest that manipulating features inside the network is more effective for adapting large frozen backbones than transforming input pixels alone. Furthermore, the strategic placement of LoRA parameters in the latter half of the vision encoder allows for high performance with significantly reduced computational overhead.

While the method shows great promise for data-efficient action detection, future work will involve exploring its generalizability to other architectures and downstream tasks like video captioning, as well as addressing the current sensitivity to hyperparameters like the number of learned augmentations.

Summary updated

Abstract

Adapting large Video-Language Models (VLMs) for action detection using only a few examples poses challenges like overfitting and the granularity mismatch between scene-level pre-training and required person-centric understanding. We propose an efficient adaptation strategy combining parameter-efficient tuning (LoRA) with a novel learnable internal feature augmentation. Applied within the frozen VLM backbone using FiLM, these augmentations generate diverse feature variations directly relevant to the task. Additionally, we introduce a group-weighted loss function that dynamically modulates the training contribution of each augmented sample based on its prediction divergence relative to the group average. This promotes robust learning by prioritizing informative yet reasonable augmentations. We demonstrate our method's effectiveness on complex multi-label, multi-person action detection datasets (AVA, MOMA), achieving strong mAP performance and showcasing significant data efficiency for adapting VLMs from limited examples.
BibTeX
@misc{patel2025group,
  author = {Deep Anil Patel and Iain Melvin and Zachary Izzo and Martin Renqiang Min},
  title = {{Group Relative Augmentation for Data Efficient Action Detection}},
  year = {2025},
  eprint = {2507.21353},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.21353},
}
Figure