Robust Zero-Shot Generalization for Open-Vocabulary Action Recognition via Task Arithmetic

Francesca Morandi, Omayma Moussadek, Federico Venturini, Mauro Suardi, Alessandro Banzatti, Francesco Cannarile, Angelo Porrello, Simone CalderaraarXiv 2026 · arXiv

TL;DR

Fine-tuning open-vocabulary action recognition models on specific target domains is often impractical due to high annotation costs, privacy regulations, and strict legal frameworks like the AI Act. This paper introduces a privacy-preserving alternative: using task arithmetic to merge multiple models fine-tuned on diverse public datasets. The resulting unified model achieves superior zero-shot generalization on completely unseen, out-of-distribution target domains without ever looking at target-domain data.

What Problem Does This Paper Address?

Open-Vocabulary Action Recognition (OVAR) allows machine learning models to identify novel human actions in video by matching visual features with textual descriptions (e.g., using CLIP-style embeddings). While powerful, deploying these models in complex real-world environments (like low-light surveillance or industrial safety monitoring) usually requires domain-specific fine-tuning to be reliable.

However, collecting and annotating video data of people raises massive privacy concerns and faces strict regulatory hurdles (such as the EU AI Act's restrictions on biometric surveillance and behavioral profiling). Furthermore, combining multiple public datasets at the data level for joint training is an engineering nightmare due to mismatched formats, sampling strategies, and label vocabularies.

The authors ask: Can we achieve robust target-domain performance without collecting target data, and without the headaches of joint training?

Methods

Instead of adapting a model to a target domain using target data, the authors propose a modular strategy that operates entirely in the parameter space of the models.

  1. Base Model Initialization: They start with Open-VCLIP, a vision-language model pre-trained on Kinetics-400 that introduces lightweight spatio-temporal attention to CLIP's visual transformer.
  2. Task Vector Extraction: They independently fine-tune this base model ($\theta_0$) on several distinct, publicly available source datasets. For each dataset, they extract a task vector ($\tau$), which represents the direction and magnitude of weight updates: $$\tau = \theta_{\text{fine-tuned}} - \theta_0$$
  3. Model Merging: They combine these task vectors into a single update vector ($\tau_M$) using different algebraic strategies and add it back to the base model with a scaling factor $\alpha$ (see Figure 1).
Task addition produces a multi-task model by summing corresponding task vectors.

The authors evaluate three merging strategies:

  • Task Arithmetic (TA): Simple element-wise summation of the task vectors.
  • Task Singular Vector (TSV-M): A structured merge using Singular Value Decomposition (SVD) to extract and combine only the most informative low-rank weight update directions, reducing interference.
  • Isotropic Merging (Iso-C): Flattens the singular value spectrum of the average task vector to prevent dominant tasks from biasing the final model.

They also experiment with Hyperparameter-Diverse Model Pools, creating a "model soup" by averaging multiple checkpoints trained on the same dataset with different learning rates or augmentations before performing the multi-task merge.

Datasets

The approach is evaluated using a rigorous leave-one-dataset-out protocol across four major action recognition benchmarks:

  • Kinetics-700 (K700): Large-scale YouTube clips of daily activities.
  • UCF101: Realistic videos of sports and human-object interactions.
  • HMDB51: Cinematic clips focusing on facial and body movements.
  • XD-Violence: A massive multi-modal benchmark for anomaly and violence detection.

Key Insights and Ideas

  • The "OOD-ness" Correlation: The authors introduced a text-based metric to quantify the semantic gap (Out-Of-Distribution score) between the target datasets and the pre-training dataset (K400). They discovered that the more distant and difficult the target dataset is, the more it benefits from model merging (see Figure 3).
  • Strength in Numbers: Zero-shot performance scales positively with the number of fused source models. Merging knowledge from multiple distinct domains creates a more well-rounded, generalized action-understanding space (see Figure 4).
Performance scales as more task vectors are aggregated, particularly on out-of-distribution datasets like XD-Violence.
  • Advanced Merging Prevents Interference: While simple task addition works well when the knowledge gap is small, advanced techniques like TSV-M and Iso-C provide a clear advantage on highly out-of-distribution targets by filtering out conflicting parameter updates.

Key Results

The merged models consistently outperformed the original zero-shot baseline across different visual backbones (ViT-B/16 and ViT-L/14).

The most dramatic improvements occurred on XD-Violence, the dataset furthest from the pre-training distribution:

  • ViT-B/16: The baseline zero-shot Top-1 accuracy was 47.48%. Simple Task Arithmetic raised this to 52.64%, while TSV-M pushed it to 54.16%. Incorporating hyperparameter diversity (ALL-8 Merged) achieved 54.75%.
  • ViT-L/14: The baseline zero-shot accuracy of 48.89% was boosted to 59.67% via TSV-M, and reached a staggering 66.59% when leveraging the hyperparameter-diverse pool.

Limitations and Future Work

While the merged models excel at generalizing to unseen target domains, the authors note a trade-off: the performance of the merged model on the source datasets themselves (in-domain multi-task evaluation) is generally inferior to individual, specialized fine-tuning. This indicates that negative interference between tasks is still an issue.

Future work will likely focus on developing more sophisticated merging strategies that can preserve peak in-domain performance while maintaining these impressive zero-shot generalization gains.

Summary updated

Abstract

Open Vocabulary Action Recognition (OVAR) enables the recognition of novel actions by leveraging vision-language representations, overcoming the limitations of traditional closed-set approaches. However, achieving robust performance in real-world scenarios typically requires domain-specific fine-tuning, which is often costly and raises privacy and regulatory concerns. In this work, we propose an alternative paradigm that bypasses target-domain training and recombines knowledge from existing datasets and models. Leveraging model merging and task arithmetic, we extract and combine task vectors from models fine-tuned on diverse public OVAR datasets. We show that, in out-of-distribution settings, the resulting merged model achieves superior zero-shot generalization to the pre-trained base model. Code is available at https://github.com/omaymaMoussadek/robust-ovar
BibTeX
@misc{morandi2026robust,
  author = {Francesca Morandi and Omayma Moussadek and Federico Venturini and Mauro Suardi and Alessandro Banzatti and Francesco Cannarile and Angelo Porrello and Simone Calderara},
  title = {{Robust Zero-Shot Generalization for Open-Vocabulary Action Recognition via Task Arithmetic}},
  year = {2026},
  eprint = {2606.20734},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.20734},
}
Figure