Ego-VPA: Egocentric Video Understanding with Parameter-efficient Adaptation

Tz-Ying Wu, Kyle Min, Subarna Tripathi, Nuno Vasconcelos

TL;DR

Ego-VPA is a parameter-efficient adaptation method for egocentric video foundation models that uses a shared "prompt basis" to synthesize context-aware prompts. By updating just 0.84% of the model's parameters, it achieves performance that matches or even exceeds full fine-tuning on complex first-person video tasks.

What Problem Does This Paper Address?

Video Foundation Models (VFMs) have revolutionized video understanding, but adapting them to specific domains—like egocentric (first-person) video—remains a challenge. Egocentric video is uniquely difficult due to rapid camera motion, significant motion blur, and frequent hand-object interactions that differ drastically from standard third-person "exocentric" footage.

While full fine-tuning can bridge this domain gap, it is computationally expensive and requires storing a full set of parameters for every new task. Existing "parameter-efficient" methods like simple prompt tuning or adapters often struggle to capture the complex temporal and cross-modal (video-text) context necessary for first-person perspectives. The authors identify a significant "zero-shot to fine-tuning" gap in current egocentric VFMs, which they aim to close without the heavy overhead of full model updates.

Methods

The core innovation is Ego-VPA (Egocentric Video Parameter-efficient Adaptation). Instead of learning a static set of prompts, it learns a prompt basis—a set of orthogonal vectors that represent the "principal components" of the prompt space.

Key components include:

  • Local Sparse Approximation: For every video frame or text feature, the model projects it into a latent space and identifies the top-$k$ most similar vectors from the shared prompt basis.
  • Prompt Synthesis: These selected basis vectors are then passed through a lightweight linear decoder to synthesize specific prompts for that particular input. This process is illustrated in the figure below:
Figure 3. Prompt Synthesis. Token z is projected into the subspace by h(·), and sparsely approximated by the top-k similar prompts in the prompt basis F, which are finally mapped into k prompts by the mapping g(·). (h,g) can be (hvid,gvid) or (htxt,gtxt) for visual or text prompt generation, respectively. k=4 in this illustration.
  • Cross-Modal Sharing: Crucially, the prompt basis is shared between the video and text encoders. This allows the model to transfer semantic knowledge between modalities efficiently.
  • Context Modeling: Because the prompts are synthesized based on frame features, they naturally capture temporal context across the video without needing heavy recurrent modules like LSTMs.

The overall architecture integrates these synthesized prompts into the spatial attention blocks of a frozen VFM (like LaViLa), as shown here:

Ego-VPA Architecture: The model leverages a shared basis F for cross-modal prompt synthesis, keeping the main VFM backbone frozen.

Datasets

The authors evaluated Ego-VPA on three major egocentric video benchmarks:

  1. Charades-Ego: A dataset featuring paired first-person and third-person views of daily activities, used here for fine-grained action classification.
  2. EGTEA (Gaze-Enabled Egocentric Action): A cooking-focused dataset used to test action classification accuracy.
  3. EPIC-Kitchens-100: A large-scale dataset used to evaluate the model's performance on Multi-Instance Retrieval (MIR) tasks.

The base model (LaViLa) was originally pre-trained on the massive Ego4D dataset.

Key Insights and Ideas

  • The "Ego" Domain Gap is Real: The authors show that even powerful models trained on third-person video (Exo-VFMs) perform poorly on egocentric tasks, even with adaptation. Specialized Ego-VFMs are better but still require fine-tuning to reach their full potential on new datasets.
  • Shared Semantics: Since a video and its description refer to the same event, using a shared prompt basis for both modalities is highly effective. It forces the model to learn a unified semantic space for prompts.
  • Sparsity is Efficiency: You don't need thousands of prompts. By selecting only the most relevant $k$ basis vectors (local subspace approximation), the model stays extremely lightweight while remaining expressive.
  • Distributed Learning: During training, the authors use a sampling strategy to ensure all vectors in the prompt basis are updated, preventing the model from over-relying on a few "popular" basis vectors early on.

Key Results

Ego-VPA consistently outperformed state-of-the-art prompt-tuning methods while using significantly fewer parameters.

  • Performance vs. Efficiency: On Charades-Ego, Ego-VPA achieved 33.8 mAP with only 0.84% tunable parameters. This actually beat the full fine-tuning result of 32.9/33.7 mAP (which uses 100% of parameters).
  • Comparison to Baselines: It outperformed the previous best baseline, VoP$^{F+C}$, which requires roughly 10% of the VFM's parameters—nearly 12 times more than Ego-VPA.
  • Robustness: Ego-VPA remained effective even in low-data regimes (using only 10% or 20% of training data) and across different video temporal resolutions (4, 8, or 16 frames).

Limitations and Future Work

  • Overfitting to Pre-training: The authors note that current Ego-VFMs like LaViLa appear somewhat overfitted to the Ego4D dataset, which is why adaptation is so critical.
  • Prompt Synthesis Overhead: While the number of tunable parameters is low, there is still a small computational cost associated with the prompt synthesis mapping ($h$ and $g$ functions) during inference.
  • Future Directions: The authors suggest that exploring more sophisticated ways to model long-range temporal dependencies within this sparse prompt framework could further improve performance on complex, multi-minute egocentric activities.

Summary updated

Abstract

Video understanding typically requires fine-tuning the large backbone when adapting to new domains. In this paper, we leverage the egocentric video foundation models (Ego-VFMs) based on video-language pre-training and propose a parameter-efficient adaptation for egocentric video tasks, namely Ego-VPA. It employs a local sparse approximation for each video frame/text feature using the basis prompts, and the selected basis prompts are used to synthesize video/text prompts. Since the basis prompts are shared across frames and modalities, it models context fusion and cross-modal transfer in an efficient fashion. Experiments show that Ego-VPA excels in lightweight adaptation (with only 0.84% learnable parameters), largely improving over baselines and reaching the performance of full fine-tuning.
BibTeX
@inproceedings{wu2025ego,
  author = {Tz-Ying Wu and Kyle Min and Subarna Tripathi and Nuno Vasconcelos},
  title = {{Ego-VPA: Egocentric Video Understanding with Parameter-efficient Adaptation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00897},
}
Figure