TL;DR
EgoCaptioner is a novel framework for egocentric video captioning that pairs high-level scene overviews with fine-grained, timestamped action steps. By employing a coarse-to-fine annotation pipeline and reinforcement learning with multi-faceted rewards, it achieves state-of-the-art performance on major egocentric benchmarks, even outperforming proprietary models like Gemini 3.1 Pro in physical reasoning and temporal alignment.
What Problem Does This Paper Address?
Egocentric (first-person) video is a goldmine for embodied AI, but captioning it is notoriously difficult. Unlike third-person video, the "actor" is mostly invisible, head movements are rapid and jarring, and the camera's viewpoint changes constantly.
Current Multimodal Large Language Models (MLLMs) often fail in three key areas:
- Temporal Precision: They miss brief actions or confuse the order of events.
- Motion Interpretation: They struggle to distinguish between the wearer's actual movement and simple camera shake.
- Detail Balance: They either provide vague summaries or dense paragraphs that lack precise timing for specific actions.
Existing formats rarely combine a broad understanding of the "who, where, and why" with a second-by-second breakdown of "what" is happening.
Methods
The authors introduce EgoCaptioner, which structures captions into two levels: a video-level overview (setting the scene and intent) and detailed steps (temporally bounded actions with start/end times).
Key technical innovations include:
- Coarse-to-Fine Annotation: Instead of asking an LLM to caption a video in one go, the authors use a two-stage pipeline. First, they establish the event structure (the "skeleton"); then, they zoom into specific event crops to fill in fine-grained details. This prevents the model from getting "lost" in long sequences.
- Task-Focused Narration: The model adapts its descriptive focus based on the activity type. For example, it prioritizes hand roles during manipulation tasks but focuses on landmarks and travel direction during locomotion.
- Reinforcement Learning via GRPO: The model is refined using Group Relative Policy Optimization (GRPO) with three specialized rewards:
- Fact-based: Uses an LLM judge to verify four types of facts (Motion, Focus, Macro, and Key-moment).
- Temporal Alignment: Uses a greedy matching algorithm based on temporal Intersection over Union (tIoU) to ensure predicted timestamps match reality.
- Style: Penalizes "fragmented" descriptions (e.g., just mentioning hand movements without the object) or "redundant" ones.
- Pose-Derived Motion Facts: To ground the model in physical reality, the authors convert raw sensor data (like head rotation or body pose) into discrete "motion facts" that serve as an independent source of truth during training.
Datasets
The authors curated a massive training set of 50,000 video-instruction-caption triplets sourced from eight diverse egocentric datasets, including Ego4D, EgoHOD, and HumanPlus-1000.
Evaluation was conducted across five rigorous benchmarks:
- EgoMe & EgoHome: For caption quality and temporal segmentation.
- PhysCapBench: For physically grounded reasoning.
- Minerva-Ego & EgoTempo: For detail-oriented question answering.
Key Insights and Ideas
- Structure First, Details Later: The "aha moment" here is that single-pass captioning is a recipe for temporal confusion. By fixing the event boundaries first (coarse) and then describing the contents (fine), the model achieves much higher temporal fidelity.
- Activity-Specific "Eyes": A one-size-fits-all captioning style doesn't work for egocentric video. A social interaction requires tracking turn-taking and gestures, while a cooking task requires tracking object state changes. Tailoring the prompt focus to the activity type significantly improves saliency.
- Sensors as Ground Truth: MLLMs often hallucinate motion. Using pose-derived facts from actual sensors provides a "physical anchor" that pure vision-language models lack, leading to more accurate descriptions of spatial directions.
Key Results
EgoCaptioner sets a new bar for open-source egocentric models:
- PhysCapBench: It scored 78.84 Overall, surpassing the proprietary Gemini 3.1 Pro (75.06). It showed particularly strong gains in understanding physical mechanisms and temporal order.
- EgoHome: It achieved a 75.51 PSJ score for temporal alignment, the highest among all evaluated open-source models.
- EgoTempo: It improved accuracy by 16.8 percentage points over its base model (Qwen3.5-9B), proving that the RL and annotation pipeline effectively teach the model to "see" brief, fine-grained actions.
Limitations and Future Work
While EgoCaptioner is a significant step forward, some challenges remain:
- Computational Cost: The multi-stage annotation and RL process (using LLM-as-a-judge) is computationally intensive.
- Sensor Dependency: The "motion facts" boost requires pose or trajectory data, which isn't available for all "in-the-wild" videos.
- Long-form Coherence: While it handles videos up to 120 seconds well, maintaining perfect coherence across hour-long egocentric streams remains an open challenge for future research.
Summary updated