RefEgo: Referring Expression Comprehension Dataset from First-Person Perception of Ego4D

Shuhei Kurita, Naoki Katsura, Eri OnamiICCV 2023 · DOI · arXiv

Abstract

RefEgo is a large-scale dataset designed to advance Referring Expression Comprehension (REC) in egocentric video. Built upon the massive Ego4D collection, RefEgo features over 12,000 video clips with natural language descriptions, challenging models to track objects amidst frequent camera motion, occlusions, and high visual ambiguity. We establish baselines using MDETR and OFA, introducing a Binary Head to handle object absence. Results highlight a significant performance gap between state-of-the-art models and human benchmarks, underscoring the complexity of real-world first-person vision.

Introduction

Traditional Referring Expression Comprehension (REC) systems—designed to locate objects based on natural language queries like "the half-eaten apple on the wooden table"—primarily rely on static, web-crawled images. However, these models often fail in dynamic, first-person (egocentric) environments encountered by smart glasses or home robots.

Egocentric vision presents three primary hurdles:

  1. Frequent Motion: Constant head movement causes rapid shifts in the visual field.
  2. Vanishing Targets: Objects frequently move out of frame or become occluded, requiring models to handle "object permanence."
  3. Visual Ambiguity: Real-world environments often contain multiple similar distractors (e.g., several identical bowls), necessitating precise linguistic grounding.

RefEgo addresses these by providing a massive, "in-the-wild" dataset for temporal REC.

The RefEgo Dataset

RefEgo is derived from the Ego4D dataset, a diverse collection of daily-life egocentric videos. It comprises 12,038 video clips (over 41 hours) covering 505 object categories across 74 worldwide locations. Unlike previous datasets, RefEgo reflects real-world complexity, with an average of 3.5 objects of the same class per scene, forcing more descriptive language (e.g., "the large white bowl with broccoli inside" vs. just "the bowl").

The RefEgo pipeline: from Ego4D video clips to dense object detection and tracked object selection.

Methodology

To establish a benchmark, the researchers adapted state-of-the-art 2D REC models, specifically MDETR (transformer-based) and OFA (sequence-to-sequence).

A critical challenge in video REC is the "No-Object" problem: standard models assume the target is always present. The authors addressed this by adding a Binary Head (BH) to MDETR, trained to predict when the referred object is absent from the frame. Furthermore, they integrated ByteTrack, a tracking-by-detection algorithm, to maintain temporal consistency and smooth out "jittery" 2D detections. New metrics, mSTIoU (mean Spatio-Temporal Intersection over Union) and mIoU+n, were introduced to reward models for correctly identifying frames where the target is missing.

Key Results

The addition of the Binary Head significantly improved performance. As shown in the ROC curves, MDETR+BH achieved an AUC of 80.5% for object presence detection, a notable jump from the base MDETR's 70.7%. This indicates that knowing when you aren't looking at the target is just as important as finding it in egocentric AI.

Performance comparison of different models showing the impact of the Binary Head (BH) on object presence detection.

Despite these improvements, a massive "Human-AI Gap" remains. While human annotators achieve a score of 77.1 mSTIoU, the best model (MDETR+BH with tracking) reaches only 37.6. This gap emphasizes that RefEgo is a formidable challenge for current AI architectures.

Conclusion

RefEgo provides a rigorous benchmark for egocentric REC. While the introduction of specialized heads and temporal tracking offers clear benefits, the substantial gap between AI and human performance suggests a need for more sophisticated, end-to-end temporal models. Future work should explore architectures that ground language and track objects simultaneously to better navigate the complexities of first-person perception.

Summary updated

BibTeX
@inproceedings{kurita2023refego,
  author = {Shuhei Kurita and Naoki Katsura and Eri Onami},
  title = {{RefEgo: Referring Expression Comprehension Dataset from First-Person Perception of Ego4D}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01396},
}
Figure