Ego-Only: Egocentric Action Detection without Exocentric Pretraining

Huiyu Wang, Mitesh Kumar Singh, Lorenzo Torresani

Abstract

We present Ego-Only, the first approach that enables state-of-the-art action detection on egocentric (first-person) videos without any form of exocentric (third-person) transferring. Despite the content and appearance gap separating the two domains, large-scale exocentric transferring has been the default choice for egocentric action detection. This is because prior works found that egocentric models are difficult to train from scratch and that transferring from exocentric representations leads to improved accuracy. However, in this paper, we revisit this common belief. Motivated by the large gap separating the two domains, we propose a strategy that enables effective training of egocentric models without exocentric transferring. Our Ego-Only approach is simple. It trains the video representation with a masked autoencoder finetuned for temporal segmentation. The learned features are then fed to an off-the-shelf temporal action localization method to detect actions. We find that this renders exocentric transferring unnecessary by showing remarkably strong results achieved by this simple Ego-Only approach on three established egocentric video datasets: Ego4D, EPIC-Kitchens-100, and Charades-Ego. On both action detection and action recognition, Ego-Only outperforms previous best exocentric transferring methods that use orders of magnitude more labels. Ego-Only sets new state-of-the-art results on these datasets and benchmarks without exocentric data.

Introduction: Challenging the Transfer Paradigm

Egocentric video, captured from a first-person perspective (e.g., via head-mounted cameras), presents unique challenges compared to standard third-person (exocentric) video. There is a profound domain gap: in egocentric views, the actor is largely invisible, interactions are highly localized and close-up, and the camera motion is significantly more dynamic. Historically, the scarcity of labeled egocentric data led researchers to rely on "exocentric transferring"—pretraining models on massive third-person datasets like ImageNet or Kinetics.

The prevailing belief was that egocentric models are too difficult to train from scratch and that the broad visual knowledge from exocentric datasets is essential. This paper challenges that assumption, arguing that exocentric pretraining may introduce biases toward features (like full-body poses or background context) that are irrelevant or even detrimental to first-person action detection.

The Ego-Only Pipeline

The proposed Ego-Only approach is a streamlined three-stage pipeline designed to leverage in-domain data effectively:

  1. Self-Supervised Pretraining (MAE): A Vision Transformer (ViT) backbone is pretrained using a Masked Autoencoder (MAE) objective on unlabeled egocentric video frames. By learning to reconstruct masked pixels, the model develops a strong understanding of the visual textures and spatial structures specific to the first-person domain.
  2. Temporal Segmentation Finetuning: The backbone is then finetuned on a proxy task of per-frame temporal segmentation. This stage teaches the model to recognize fine-grained action boundaries and specific interactions using the limited labeled egocentric data available.
  3. Temporal Action Detection: Finally, the features extracted from the finetuned backbone are used by an off-the-shelf temporal action localization detector (ActionFormer). This stage focuses on the long-range temporal reasoning required to identify the start and end of actions in long-form video.
Comparison of the proposed Ego-Only pipeline versus traditional exocentric pretraining.

Experimental Results

Ego-Only was evaluated on three major benchmarks: Ego4D, EPIC-Kitchens-100, and Charades-Ego. Across all datasets, the approach achieved new state-of-the-art (SOTA) results, often by significant margins.

On the Ego4D Action Detection task, Ego-Only reached 17.9% mAP, representing a 170% relative improvement over the original baseline. Notably, it outperformed EgoVLP, a method that utilizes 4 million video-text pairs, despite Ego-Only using only 14,000 annotated segments. On EPIC-Kitchens-100, the model improved action detection by +5.5% for verbs and +6.2% for nouns.

Ego-Only performance vs. data size on Ego4D and Charades-Ego.

The results demonstrate a clear trend: in-domain pretraining with self-supervision is more powerful than massive out-of-domain supervised pretraining. As shown in the figure above, Ego-Only (represented by the green star) achieves higher mAP while using orders of magnitude less labeled data than transfer-based methods.

Conclusion

The success of Ego-Only suggests that the "Ego-Exo" gap is a fundamental barrier that is better respected than bridged. By staying strictly within the egocentric domain, models can avoid the negative transfer of irrelevant third-person features. This work highlights that data quality and domain relevance are more critical than sheer volume, paving the way for more efficient and specialized video understanding models.

Summary updated

Abstract

We present Ego-Only, the first approach that enables state-of-the-art action detection on egocentric (first-person) videos without any form of exocentric (third-person) transferring. Despite the content and appearance gap separating the two domains, large-scale exocentric transferring has been the default choice for egocentric action detection. This is because prior works found that egocentric models are difficult to train from scratch and that transferring from exocentric representations leads to improved accuracy. However, in this paper, we revisit this common belief. Motivated by the large gap separating the two domains, we propose a strategy that enables effective training of egocentric models without exocentric transferring. Our Ego-Only approach is simple. It trains the video representation with a masked autoencoder finetuned for temporal segmentation. The learned features are then fed to an off-the-shelf temporal action localization method to detect actions. We find that this renders exocentric transferring unnecessary by showing remarkably strong results achieved by this simple Ego-Only approach on three established egocentric video datasets: Ego4D, EPIC-Kitchens-100, and Charades-Ego. On both action detection and action recognition, Ego-Only outperforms previous best exocentric transferring methods that use orders of magnitude more labels. Ego-Only sets new state-of-the-art results on these datasets and benchmarks without exocentric data.
BibTeX
@inproceedings{wang2023ego,
  author = {Huiyu Wang and Mitesh Kumar Singh and Lorenzo Torresani},
  title = {{Ego-Only: Egocentric Action Detection without Exocentric Pretraining}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=joHhlGrYcU},
}
Figure