Unsupervised Ego- and Exo-centric Dense Procedural Activity Captioning via Gaze Consensus Adaptation

Zhaofeng Shi, Heqian Qiu, Lanxiao Wang, Qingbo Wu, Fanman Meng, Hongliang Li

Introduction

Humans possess a remarkable ability to watch a third-person (exocentric) tutorial and seamlessly translate those actions into a first-person (egocentric) execution. However, for artificial intelligence, this viewpoint shift remains a significant hurdle. Dense Video Captioning (DVC)—the task of localizing and describing multiple events within an untrimmed video—performs well within a single viewpoint but often fails when applied across different perspectives.

To address this, we introduce the Unsupervised Ego-Exo Dense Procedural Activity Captioning (UE$^2$DPAC) task. The objective is to train a model on a labeled source view (e.g., exocentric) and accurately predict timestamps and captions for an unlabeled target view (e.g., egocentric), eliminating the need for labor-intensive manual labeling of every new viewpoint.

The Challenge of Cross-View Adaptation

Cross-view transfer is complicated by two primary factors: temporal misalignment and visual interference. Temporal misalignment occurs because different individuals perform actions at varying speeds. Visual interference arises from the drastically different background clutter and camera angles inherent in egocentric versus exocentric recordings.

Overview of the UE$^2$DPAC task, highlighting temporal misalignment and visual interference between egocentric and exocentric views.

The GCEAN Architecture

To bridge this gap, we propose the Gaze Consensus-guided Ego-Exo Adaptation Network (GCEAN). This architecture leverages the fact that while viewpoints change, human attention remains relatively constant across the same task. GCEAN consists of two primary components designed to handle global and local domain gaps.

Score-based Adversarial Learning Module (SALM)

The SALM addresses the global domain gap by extracting view-invariant features. It utilizes a feature converter and a discriminative scoring network. By training the converter to produce features that "fool" the discriminator into being unable to distinguish between Ego and Exo views, the network learns a shared representation space.

Gaze Consensus Construction Module (GCCM)

The GCCM is the core innovation of this work. Since hardware-based eye-tracking is often unavailable, the model predicts gaze features directly from video frames. These predictions serve as an attention mechanism to calibrate representations, forcing the model to focus on active hand-object interactions while filtering out background noise. Hierarchical gaze-guided consistency losses are then applied to align temporal sequences, effectively correcting for temporal misalignment.

The Gaze Consensus-guided Ego-Exo Adaptation Network (GCEAN) architecture, featuring the SALM and GCCM modules.

Experimental Results

We evaluated GCEAN on the newly proposed EgoMe-UE$^2$DPAC dataset, which contains paired Ego and Exo videos of procedural tasks. The model was tested in two settings: Exo2Ego (training on exocentric, testing on egocentric) and Ego2Exo (the reverse).

  • Exo2Ego Performance: GCEAN achieved a CIDEr score of 40.78, significantly outperforming existing state-of-the-art methods like Sync (38.53) and Exo2EgoDVC (35.44).
  • Ego2Exo Performance: The model reached a CIDEr score of 38.61, maintaining a clear lead over baselines.
  • Cross-Dataset Robustness: When trained on EgoMe-UE$^2$DPAC and tested on the EgoYC2 dataset, GCEAN demonstrated superior generalization, achieving a 4.63 CIDEr score compared to 3.10 for the nearest baseline.

Conclusion

GCEAN demonstrates that "gaze consensus" is a powerful tool for cross-view adaptation. By anchoring model attention to human visual focus, we can achieve robust dense video captioning across egocentric and exocentric perspectives without target-view annotations. Future work will explore meta-learning strategies to automate hyperparameter optimization and further improve the efficiency of these complex adaptation networks.

Summary updated

Abstract

Even from an early age, humans naturally adapt between exocentric (Exo) and egocentric (Ego) perspectives to understand daily procedural activities. Inspired by this cognitive ability, we propose a novel Unsupervised Ego-Exo Dense Procedural Activity Captioning (UE$^{2}$DPAC) task, which aims to transfer knowledge from the labeled source view to predict the time segments and descriptions of action sequences for the target view without annotations. Despite previous works endeavoring to address the fully-supervised single-view or cross-view dense video captioning, they lapse in the proposed task due to the significant inter-view gap caused by temporal misalignment and irrelevant object interference. Hence, we propose a Gaze Consensus-guided Ego-Exo Adaptation Network (GCEAN) that injects the gaze information into the learned representations for the fine-grained Ego-Exo alignment. Specifically, we propose a Score-based Adversarial Learning Module (SALM) that incorporates a discriminative scoring network and compares the scores of distinct views to learn unified view-invariant representations from a global level. Then, the Gaze Consensus Construction Module (GCCM) utilizes the gaze to progressively calibrate the learned representations to highlight the regions of interest and extract the corresponding temporal contexts. Moreover, we adopt hierarchical gaze-guided consistency losses to construct gaze consensus for the explicit temporal and spatial adaptation between the source and target views. To support our research, we propose a new EgoMe-UE$^{2}$DPAC benchmark, and extensive experiments demonstrate the effectiveness of our method, which outperforms many related methods by a large margin. Code is available at https://github.com/ZhaofengSHI/GCEAN.
BibTeX
@inproceedings{shi2025unsupervised,
  author = {Zhaofeng Shi and Heqian Qiu and Lanxiao Wang and Qingbo Wu and Fanman Meng and Hongliang Li},
  title = {{Unsupervised Ego- and Exo-centric Dense Procedural Activity Captioning via Gaze Consensus Adaptation}},
  booktitle = {ACM Multimedia},
  year = {2025},
  doi = {10.1145/3746027.3755167},
}
Figure