VIOLA: Towards Video In-Context Learning with Minimal Annotations

Ryo Fujii, Hideo Saito, Ryo HachiumaarXiv 2026 · arXiv

TL;DR

VIOLA is a new framework that enables Multimodal Large Language Models (MLLMs) to adapt to specialized video domains (like surgery or industrial work) using only a handful of human-annotated examples. By intelligently selecting which samples to label and leveraging unlabeled data through "confidence-aware" pseudo-labeling, VIOLA achieves massive performance gains over standard zero-shot and few-shot methods.

What Problem Does This Paper Address?

While Multimodal Large Language Models (MLLMs) like Qwen2-VL or LLaVA are impressive, they often struggle when dropped into specialized environments—think of a model trying to identify surgical phases or industrial assembly steps.

The standard way to fix this is In-Context Learning (ICL), where you provide the model with a few "demonstration" examples in the prompt. However, existing ICL methods assume you already have a massive pool of labeled videos to choose from. In the real world, getting an expert surgeon or a senior engineer to label hundreds of videos is prohibitively expensive.

The authors identify three main hurdles:

  1. Selection Inefficacy: Standard ways of picking samples to label often choose "outliers" (weird, unrepresentative videos) instead of useful ones.
  2. Hybrid Noise: If you use a model to "pseudo-label" unlabeled data to expand your pool, the errors in those labels can confuse the MLLM.
  3. Sensitivity: Video models are extremely sensitive to the quality of their context; one bad example can derail the entire reasoning process.

Methods

The authors introduce VIOLA (Video In-cOntext Learning with minimal Annotation), which operates in three distinct stages:

1. Density-Uncertainty-Weighted Selection

Instead of picking samples for human annotation at random, VIOLA uses a "stratified" approach. It uses a Gaussian Mixture Model (GMM) to cluster the video embeddings (density) and checks the model's zero-shot confidence (uncertainty). By balancing these, it picks samples that are diverse (covering different clusters), representative (near the center of a cluster), and informative (where the model is currently confused).

2. In-Context Pseudo-Annotation

Once the tiny human-labeled set is ready, VIOLA uses it to label the remaining unlabeled data. Crucially, it only keeps the pseudo-labels where the model is highly confident (top 5th percentile), creating a "Hybrid Pool" of high-quality gold labels and silver pseudo-labels.

3. Confidence-Aware Inference

When it’s time to actually use the model, VIOLA doesn't just treat all demonstrations as equal.

  • Retrieval: It retrieves examples based on a composite score of visual similarity and label confidence.
  • Prompting: It explicitly tells the MLLM the source and reliability of the example. For instance: "The predicted label is [X] (Pseudo-label with 90% confidence)." This acts as a "soft-gate," allowing the model to prioritize human-verified patterns over noisy ones.

Datasets

The researchers tested VIOLA across nine diverse benchmarks to prove it works in "the wild":

  • Classification: Drive&Act (driving), EgoPet (animal behavior), EgoSurgery (medical), ENIGMA (industrial), UCF-Crime (surveillance), Xsports (extreme sports), and MammAlps (wildlife).
  • Captioning: Bora (biomedical) and CapERA (aerial surveillance).

These datasets represent specialized domains where general-purpose pre-training usually falls short.

Key Insights and Ideas

  • The Outlier Trap: The paper highlights that standard active learning often fails in video because "uncertain" samples are often just noisy background footage. By adding a density weight, VIOLA ensures the model learns from "useful difficulty" rather than just visual noise.
  • Communication is Key: One of the most elegant parts of the paper is the Confidence-Aware Prompting. Simply telling the model "we think this is the label, but we're only 70% sure" significantly improves the model's ability to filter out bad demonstrations during its own reasoning process.
  • Synergy over Isolation: The paper proves that selective annotation and pseudo-labeling are better together. The human labels provide the "anchor," and the pseudo-labels provide the "scale."

Key Results

Using a tiny budget of just 20 labeled samples:

  • Massive Gains: On the industrial ENIGMA dataset, VIOLA improved Qwen2-VL's performance by +53.6% over the zero-shot baseline.
  • Efficiency: In many cases, VIOLA achieved the same accuracy with 20x fewer annotations than "Naive ICL" (random selection).
  • Consistency: The framework worked across four different MLLM architectures (Qwen2-VL, VideoLLaMA3, Qwen3-VL, and LLaVA-Video), showing it is a general-purpose strategy, not a model-specific hack.
  • Scalability: Unlike some methods that plateau, VIOLA’s performance continued to climb steadily as the annotation budget increased from 20 to 100 samples.

Limitations and Future Work

  • Embedding Reliance: VIOLA relies on pre-trained visual embeddings (like InternVideo2) to perform its initial clustering. If a domain is so specialized that the encoder can't even "see" the relevant features, the selection process might struggle.
  • Computational Cost: The iterative retrieval and pseudo-labeling process adds some overhead, though the authors note it is still much cheaper than full model fine-tuning.
  • Future Directions: The authors suggest exploring domain-adapted encoders that could further refine the semantic space for even more accurate sample selection in extreme edge cases.

Summary updated

Abstract

Generalizing Multimodal Large Language Models (MLLMs) to novel video domains is essential for real-world deployment but remains challenging due to the scarcity of labeled data. While In-Context Learning (ICL) offers a training-free adaptation path, standard methods rely on large annotated pools, which are often impractical in specialized environments like industrial or surgical settings since they require the experts' annotations. To bridge this gap, we introduce VIOLA (Video In-cOntext Learning with minimal Annotation), a label-efficient framework that synergizes minimal expert supervision with abundant unlabeled data. First, to maximize the efficiency of a strict annotation budget, we propose density-uncertainty-weighted sampling. Unlike standard diversity or uncertainty strategies that risk selecting visual outliers, our method leverages density estimation to identify samples that are simultaneously diverse, representative, and informative. Second, to utilize the remaining unlabeled data without noise propagation, we construct a hybrid pool and introduce confidence-aware retrieval and confidence-aware prompting. These mechanisms explicitly model label reliability, retrieving demonstrations based on a composite score of similarity and confidence while enabling the MLLM to adaptively distinguish between verified ground truths and noisy pseudo-labels. Extensive experiments across nine diverse benchmarks using four MLLMs demonstrate that our framework significantly outperforms various baselines in low-resource settings, achieving robust adaptation with minimal annotation costs.
BibTeX
@misc{fujii2026viola,
  author = {Ryo Fujii and Hideo Saito and Ryo Hachiuma},
  title = {{VIOLA: Towards Video In-Context Learning with Minimal Annotations}},
  year = {2026},
  eprint = {2601.15549},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.15549},
  howpublished = {arXiv},
}
Figure