Abstract
RefEgo is a large-scale dataset designed to advance Referring Expression Comprehension (REC) in egocentric video. Built upon the massive Ego4D collection, RefEgo features over 12,000 video clips with natural language descriptions, challenging models to track objects amidst frequent camera motion, occlusions, and high visual ambiguity. We establish baselines using MDETR and OFA, introducing a Binary Head to handle object absence. Results highlight a significant performance gap between state-of-the-art models and human benchmarks, underscoring the complexity of real-world first-person vision.
Introduction
Traditional Referring Expression Comprehension (REC) systems—designed to locate objects based on natural language queries like "the half-eaten apple on the wooden table"—primarily rely on static, web-crawled images. However, these models often fail in dynamic, first-person (egocentric) environments encountered by smart glasses or home robots.
Egocentric vision presents three primary hurdles:
- Frequent Motion: Constant head movement causes rapid shifts in the visual field.
- Vanishing Targets: Objects frequently move out of frame or become occluded, requiring models to handle "object permanence."
- Visual Ambiguity: Real-world environments often contain multiple similar distractors (e.g., several identical bowls), necessitating precise linguistic grounding.
RefEgo addresses these by providing a massive, "in-the-wild" dataset for temporal REC.
The RefEgo Dataset
RefEgo is derived from the Ego4D dataset, a diverse collection of daily-life egocentric videos. It comprises 12,038 video clips (over 41 hours) covering 505 object categories across 74 worldwide locations. Unlike previous datasets, RefEgo reflects real-world complexity, with an average of 3.5 objects of the same class per scene, forcing more descriptive language (e.g., "the large white bowl with broccoli inside" vs. just "the bowl").
Methodology
To establish a benchmark, the researchers adapted state-of-the-art 2D REC models, specifically MDETR (transformer-based) and OFA (sequence-to-sequence).
A critical challenge in video REC is the "No-Object" problem: standard models assume the target is always present. The authors addressed this by adding a Binary Head (BH) to MDETR, trained to predict when the referred object is absent from the frame. Furthermore, they integrated ByteTrack, a tracking-by-detection algorithm, to maintain temporal consistency and smooth out "jittery" 2D detections. New metrics, mSTIoU (mean Spatio-Temporal Intersection over Union) and mIoU+n, were introduced to reward models for correctly identifying frames where the target is missing.
Key Results
The addition of the Binary Head significantly improved performance. As shown in the ROC curves, MDETR+BH achieved an AUC of 80.5% for object presence detection, a notable jump from the base MDETR's 70.7%. This indicates that knowing when you aren't looking at the target is just as important as finding it in egocentric AI.
Despite these improvements, a massive "Human-AI Gap" remains. While human annotators achieve a score of 77.1 mSTIoU, the best model (MDETR+BH with tracking) reaches only 37.6. This gap emphasizes that RefEgo is a formidable challenge for current AI architectures.
Conclusion
RefEgo provides a rigorous benchmark for egocentric REC. While the introduction of specialized heads and temporal tracking offers clear benefits, the substantial gap between AI and human performance suggests a need for more sophisticated, end-to-end temporal models. Future work should explore architectures that ground language and track objects simultaneously to better navigate the complexities of first-person perception.
Summary updated