Abstract
We present Ego-Only, the first approach that enables state-of-the-art action detection on egocentric (first-person) videos without any form of exocentric (third-person) transferring. Despite the content and appearance gap separating the two domains, large-scale exocentric transferring has been the default choice for egocentric action detection. This is because prior works found that egocentric models are difficult to train from scratch and that transferring from exocentric representations leads to improved accuracy. However, in this paper, we revisit this common belief. Motivated by the large gap separating the two domains, we propose a strategy that enables effective training of egocentric models without exocentric transferring. Our Ego-Only approach is simple. It trains the video representation with a masked autoencoder finetuned for temporal segmentation. The learned features are then fed to an off-the-shelf temporal action localization method to detect actions. We find that this renders exocentric transferring unnecessary by showing remarkably strong results achieved by this simple Ego-Only approach on three established egocentric video datasets: Ego4D, EPIC-Kitchens-100, and Charades-Ego. On both action detection and action recognition, Ego-Only outperforms previous best exocentric transferring methods that use orders of magnitude more labels. Ego-Only sets new state-of-the-art results on these datasets and benchmarks without exocentric data.
Introduction: Challenging the Status Quo
In computer vision, "egocentric" video—captured from head-mounted cameras—is notoriously difficult to process. For years, the standard practice has been to pretrain models on massive "exocentric" (third-person) datasets like Kinetics or ImageNet and then transfer that knowledge to first-person tasks. This reliance on exocentric transferring stems from the belief that egocentric models are too difficult to train from scratch due to data scarcity and the inherent complexity of first-person perspectives.
However, this "exocentric transferring" is a double-edged sword. A massive domain gap exists between the two perspectives:
- Perspective: Exocentric videos show the actor from a distance; egocentric videos show only the actor's hands and close-up object interactions.
- Duration: Third-person clips are usually short and "trimmed," while first-person videos are long-form and uncurated.
- Granularity: First-person tasks require distinguishing between very similar actions, such as "wiping a counter" versus "wiping an appliance."
The authors of "Ego-Only" argue that this domain gap can introduce detrimental biases. With the advent of large-scale egocentric datasets like Ego4D and efficient self-supervised learning techniques, they revisit the necessity of third-person pretraining.
The Ego-Only Pipeline
The proposed "Ego-Only" approach is a simple three-stage pipeline that avoids exocentric data entirely, focusing solely on in-domain learning.
The pipeline consists of:
- Masked Autoencoder (MAE) Pretraining: A Vision Transformer (ViT) backbone learns to reconstruct missing patches of video frames. This allows the model to learn the visual distribution of egocentric video without human labels.
- Temporal Semantic Segmentation Finetuning: The model is finetuned on a "proxy" task: predicting action labels for every frame in a clip. This teaches the model to distinguish fine-grained actions and handle the long-tail distribution of real-world activities.
- Temporal Action Localization: Finally, features from the finetuned backbone are frozen and fed into an off-the-shelf detector (ActionFormer) to locate actions across long-form video.
Experimental Results
The researchers evaluated Ego-Only on three major egocentric benchmarks: Ego4D, EPIC-Kitchens-100, and Charades-Ego. The results consistently showed that Ego-Only outperforms methods relying on massive exocentric transferring.
Key findings include:
- Superior Accuracy: On Ego4D Action Detection, Ego-Only achieved 17.9% mAP, significantly higher than the previous best (EgoVLP at 11.39%).
- Data Efficiency: Ego-Only achieved better results using only 14K egocentric labels compared to previous state-of-the-art methods that used over 18 million exocentric labels. This suggests that the relevance of data (in-domain) is more important than the sheer quantity of out-of-domain data.
- The Importance of MAE: Ablation studies showed that skipping the MAE pretraining stage caused a massive performance drop (from 16.3% mAP to 4.2% on Ego4D), proving that self-supervision is critical for training from scratch.
Conclusion
Ego-Only proves that exocentric data is not a prerequisite for high-performance egocentric action detection. By leveraging in-domain self-supervision and a simple temporal segmentation proxy task, the model avoids the biases inherent in third-person datasets. While exocentric data might still offer value, this work demonstrates that a focused, "ego-only" approach is currently the most effective path forward for first-person vision tasks.
Summary updated