TL;DR
Fine-tuning open-vocabulary action recognition models on specific target domains is often impractical due to high annotation costs, privacy regulations, and strict legal frameworks like the AI Act. This paper introduces a privacy-preserving alternative: using task arithmetic to merge multiple models fine-tuned on diverse public datasets. The resulting unified model achieves superior zero-shot generalization on completely unseen, out-of-distribution target domains without ever looking at target-domain data.
What Problem Does This Paper Address?
Open-Vocabulary Action Recognition (OVAR) allows machine learning models to identify novel human actions in video by matching visual features with textual descriptions (e.g., using CLIP-style embeddings). While powerful, deploying these models in complex real-world environments (like low-light surveillance or industrial safety monitoring) usually requires domain-specific fine-tuning to be reliable.
However, collecting and annotating video data of people raises massive privacy concerns and faces strict regulatory hurdles (such as the EU AI Act's restrictions on biometric surveillance and behavioral profiling). Furthermore, combining multiple public datasets at the data level for joint training is an engineering nightmare due to mismatched formats, sampling strategies, and label vocabularies.
The authors ask: Can we achieve robust target-domain performance without collecting target data, and without the headaches of joint training?
Methods
Instead of adapting a model to a target domain using target data, the authors propose a modular strategy that operates entirely in the parameter space of the models.
- Base Model Initialization: They start with Open-VCLIP, a vision-language model pre-trained on Kinetics-400 that introduces lightweight spatio-temporal attention to CLIP's visual transformer.
- Task Vector Extraction: They independently fine-tune this base model ($\theta_0$) on several distinct, publicly available source datasets. For each dataset, they extract a task vector ($\tau$), which represents the direction and magnitude of weight updates: $$\tau = \theta_{\text{fine-tuned}} - \theta_0$$
- Model Merging: They combine these task vectors into a single update vector ($\tau_M$) using different algebraic strategies and add it back to the base model with a scaling factor $\alpha$ (see Figure 1).
The authors evaluate three merging strategies:
- Task Arithmetic (TA): Simple element-wise summation of the task vectors.
- Task Singular Vector (TSV-M): A structured merge using Singular Value Decomposition (SVD) to extract and combine only the most informative low-rank weight update directions, reducing interference.
- Isotropic Merging (Iso-C): Flattens the singular value spectrum of the average task vector to prevent dominant tasks from biasing the final model.
They also experiment with Hyperparameter-Diverse Model Pools, creating a "model soup" by averaging multiple checkpoints trained on the same dataset with different learning rates or augmentations before performing the multi-task merge.
Datasets
The approach is evaluated using a rigorous leave-one-dataset-out protocol across four major action recognition benchmarks:
- Kinetics-700 (K700): Large-scale YouTube clips of daily activities.
- UCF101: Realistic videos of sports and human-object interactions.
- HMDB51: Cinematic clips focusing on facial and body movements.
- XD-Violence: A massive multi-modal benchmark for anomaly and violence detection.
Key Insights and Ideas
- The "OOD-ness" Correlation: The authors introduced a text-based metric to quantify the semantic gap (Out-Of-Distribution score) between the target datasets and the pre-training dataset (K400). They discovered that the more distant and difficult the target dataset is, the more it benefits from model merging (see Figure 3).
- Strength in Numbers: Zero-shot performance scales positively with the number of fused source models. Merging knowledge from multiple distinct domains creates a more well-rounded, generalized action-understanding space (see Figure 4).
- Advanced Merging Prevents Interference: While simple task addition works well when the knowledge gap is small, advanced techniques like TSV-M and Iso-C provide a clear advantage on highly out-of-distribution targets by filtering out conflicting parameter updates.
Key Results
The merged models consistently outperformed the original zero-shot baseline across different visual backbones (ViT-B/16 and ViT-L/14).
The most dramatic improvements occurred on XD-Violence, the dataset furthest from the pre-training distribution:
- ViT-B/16: The baseline zero-shot Top-1 accuracy was 47.48%. Simple Task Arithmetic raised this to 52.64%, while TSV-M pushed it to 54.16%. Incorporating hyperparameter diversity (ALL-8 Merged) achieved 54.75%.
- ViT-L/14: The baseline zero-shot accuracy of 48.89% was boosted to 59.67% via TSV-M, and reached a staggering 66.59% when leveraging the hyperparameter-diverse pool.
Limitations and Future Work
While the merged models excel at generalizing to unseen target domains, the authors note a trade-off: the performance of the merged model on the source datasets themselves (in-domain multi-task evaluation) is generally inferior to individual, specialized fine-tuning. This indicates that negative interference between tasks is still an issue.
Future work will likely focus on developing more sophisticated merging strategies that can preserve peak in-domain performance while maintaining these impressive zero-shot generalization gains.
Summary updated