Introduction
Humans possess a remarkable ability to watch a third-person (exocentric) tutorial and seamlessly translate those actions into a first-person (egocentric) execution. However, for artificial intelligence, this viewpoint shift remains a significant hurdle. Dense Video Captioning (DVC)—the task of localizing and describing multiple events within an untrimmed video—performs well within a single viewpoint but often fails when applied across different perspectives.
To address this, we introduce the Unsupervised Ego-Exo Dense Procedural Activity Captioning (UE$^2$DPAC) task. The objective is to train a model on a labeled source view (e.g., exocentric) and accurately predict timestamps and captions for an unlabeled target view (e.g., egocentric), eliminating the need for labor-intensive manual labeling of every new viewpoint.
The Challenge of Cross-View Adaptation
Cross-view transfer is complicated by two primary factors: temporal misalignment and visual interference. Temporal misalignment occurs because different individuals perform actions at varying speeds. Visual interference arises from the drastically different background clutter and camera angles inherent in egocentric versus exocentric recordings.
The GCEAN Architecture
To bridge this gap, we propose the Gaze Consensus-guided Ego-Exo Adaptation Network (GCEAN). This architecture leverages the fact that while viewpoints change, human attention remains relatively constant across the same task. GCEAN consists of two primary components designed to handle global and local domain gaps.
Score-based Adversarial Learning Module (SALM)
The SALM addresses the global domain gap by extracting view-invariant features. It utilizes a feature converter and a discriminative scoring network. By training the converter to produce features that "fool" the discriminator into being unable to distinguish between Ego and Exo views, the network learns a shared representation space.
Gaze Consensus Construction Module (GCCM)
The GCCM is the core innovation of this work. Since hardware-based eye-tracking is often unavailable, the model predicts gaze features directly from video frames. These predictions serve as an attention mechanism to calibrate representations, forcing the model to focus on active hand-object interactions while filtering out background noise. Hierarchical gaze-guided consistency losses are then applied to align temporal sequences, effectively correcting for temporal misalignment.
Experimental Results
We evaluated GCEAN on the newly proposed EgoMe-UE$^2$DPAC dataset, which contains paired Ego and Exo videos of procedural tasks. The model was tested in two settings: Exo2Ego (training on exocentric, testing on egocentric) and Ego2Exo (the reverse).
- Exo2Ego Performance: GCEAN achieved a CIDEr score of 40.78, significantly outperforming existing state-of-the-art methods like
Sync(38.53) andExo2EgoDVC(35.44). - Ego2Exo Performance: The model reached a CIDEr score of 38.61, maintaining a clear lead over baselines.
- Cross-Dataset Robustness: When trained on EgoMe-UE$^2$DPAC and tested on the EgoYC2 dataset, GCEAN demonstrated superior generalization, achieving a 4.63 CIDEr score compared to 3.10 for the nearest baseline.
Conclusion
GCEAN demonstrates that "gaze consensus" is a powerful tool for cross-view adaptation. By anchoring model attention to human visual focus, we can achieve robust dense video captioning across egocentric and exocentric perspectives without target-view annotations. Future work will explore meta-learning strategies to automate hyperparameter optimization and further improve the efficiency of these complex adaptation networks.
Summary updated