EgoCaptioner: Towards Comprehensive and Fine-Grained Egocentric Video Captioning

Yuqi Tang, Yang Shi, Yuran Wang, Tengfei Liu, Rundong Wang, Zongyang Qiu, Qixun Wang, Zhuoran Zhang, Xuanyu Zhu, Yifan Xu, Yiyan Ji, Shujie Li, Shixuan Liu, Haotian Wang, Yuanxing Zhang, Pengfei Wan, Chengwei QinarXiv 2026 · arXiv

TL;DR

EgoCaptioner is a novel framework for egocentric video captioning that pairs high-level scene overviews with fine-grained, timestamped action steps. By employing a coarse-to-fine annotation pipeline and reinforcement learning with multi-faceted rewards, it achieves state-of-the-art performance on major egocentric benchmarks, even outperforming proprietary models like Gemini 3.1 Pro in physical reasoning and temporal alignment.

What Problem Does This Paper Address?

Egocentric (first-person) video is a goldmine for embodied AI, but captioning it is notoriously difficult. Unlike third-person video, the "actor" is mostly invisible, head movements are rapid and jarring, and the camera's viewpoint changes constantly.

Current Multimodal Large Language Models (MLLMs) often fail in three key areas:

  1. Temporal Precision: They miss brief actions or confuse the order of events.
  2. Motion Interpretation: They struggle to distinguish between the wearer's actual movement and simple camera shake.
  3. Detail Balance: They either provide vague summaries or dense paragraphs that lack precise timing for specific actions.

Existing formats rarely combine a broad understanding of the "who, where, and why" with a second-by-second breakdown of "what" is happening.

Methods

The authors introduce EgoCaptioner, which structures captions into two levels: a video-level overview (setting the scene and intent) and detailed steps (temporally bounded actions with start/end times).

Key technical innovations include:

  • Coarse-to-Fine Annotation: Instead of asking an LLM to caption a video in one go, the authors use a two-stage pipeline. First, they establish the event structure (the "skeleton"); then, they zoom into specific event crops to fill in fine-grained details. This prevents the model from getting "lost" in long sequences.
  • Task-Focused Narration: The model adapts its descriptive focus based on the activity type. For example, it prioritizes hand roles during manipulation tasks but focuses on landmarks and travel direction during locomotion.
  • Reinforcement Learning via GRPO: The model is refined using Group Relative Policy Optimization (GRPO) with three specialized rewards:
  • Fact-based: Uses an LLM judge to verify four types of facts (Motion, Focus, Macro, and Key-moment).
  • Temporal Alignment: Uses a greedy matching algorithm based on temporal Intersection over Union (tIoU) to ensure predicted timestamps match reality.
  • Style: Penalizes "fragmented" descriptions (e.g., just mentioning hand movements without the object) or "redundant" ones.
  • Pose-Derived Motion Facts: To ground the model in physical reality, the authors convert raw sensor data (like head rotation or body pose) into discrete "motion facts" that serve as an independent source of truth during training.
The EgoCaptioner framework produces a two-level caption: a global overview and a sequence of detailed, timestamped steps.

Datasets

The authors curated a massive training set of 50,000 video-instruction-caption triplets sourced from eight diverse egocentric datasets, including Ego4D, EgoHOD, and HumanPlus-1000.

Evaluation was conducted across five rigorous benchmarks:

  • EgoMe & EgoHome: For caption quality and temporal segmentation.
  • PhysCapBench: For physically grounded reasoning.
  • Minerva-Ego & EgoTempo: For detail-oriented question answering.

Key Insights and Ideas

  • Structure First, Details Later: The "aha moment" here is that single-pass captioning is a recipe for temporal confusion. By fixing the event boundaries first (coarse) and then describing the contents (fine), the model achieves much higher temporal fidelity.
  • Activity-Specific "Eyes": A one-size-fits-all captioning style doesn't work for egocentric video. A social interaction requires tracking turn-taking and gestures, while a cooking task requires tracking object state changes. Tailoring the prompt focus to the activity type significantly improves saliency.
  • Sensors as Ground Truth: MLLMs often hallucinate motion. Using pose-derived facts from actual sensors provides a "physical anchor" that pure vision-language models lack, leading to more accurate descriptions of spatial directions.
The coarse-to-fine pipeline ensures that event boundaries are established before the model attempts to narrate fine-grained details.

Key Results

EgoCaptioner sets a new bar for open-source egocentric models:

  • PhysCapBench: It scored 78.84 Overall, surpassing the proprietary Gemini 3.1 Pro (75.06). It showed particularly strong gains in understanding physical mechanisms and temporal order.
  • EgoHome: It achieved a 75.51 PSJ score for temporal alignment, the highest among all evaluated open-source models.
  • EgoTempo: It improved accuracy by 16.8 percentage points over its base model (Qwen3.5-9B), proving that the RL and annotation pipeline effectively teach the model to "see" brief, fine-grained actions.
Comparison on PhysCapBench shows EgoCaptioner leading in every dimension, especially in temporal order (TO) and physical mechanisms (MC).

Limitations and Future Work

While EgoCaptioner is a significant step forward, some challenges remain:

  • Computational Cost: The multi-stage annotation and RL process (using LLM-as-a-judge) is computationally intensive.
  • Sensor Dependency: The "motion facts" boost requires pose or trajectory data, which isn't available for all "in-the-wild" videos.
  • Long-form Coherence: While it handles videos up to 120 seconds well, maintaining perfect coherence across hour-long egocentric streams remains an open challenge for future research.

Summary updated

Abstract

Egocentric video captioning requires fine-grained detail and accurate timing, yet current multimodal large language models (MLLMs) often miss brief actions, confuse their order, or misinterpret the wearer's motion. Existing caption formats also offer limited support for combining detailed narration with precise timing. We present EgoCaptioner, which pairs a video-level overview of the setting, participants, and the wearer's overall activity with timed steps whose narration adapts to varying perceptual demands and marks key moments at specific timestamps. We develop a coarse-to-fine annotation pipeline that establishes event structure before expanding step descriptions, yielding 50,000 training examples from eight egocentric video sources. Complementary rewards assess factual accuracy and coverage, measure step-level temporal alignment, and penalize fragmented or redundant narration, guiding caption refinement through Group Relative Policy Optimization (GRPO). To improve the reliability of factual supervision, we supplement caption references with pose-derived motion facts when sensor data are available. Across five egocentric benchmarks, EgoCaptioner achieves leading performance among evaluated open-source models in caption quality, caption-conditioned question answering, and temporal alignment. It also surpasses Gemini 3.1 Pro on PhysCapBench and all evaluated proprietary baselines on Minerva-Ego under the caption-to-QA protocol. Ablations support the complementary roles of factual and temporal rewards and show that coarse-to-fine annotation provides more effective supervision than single-pass annotation.
BibTeX
@misc{tang2026egocaptioner,
  author = {Yuqi Tang and Yang Shi and Yuran Wang and Tengfei Liu and Rundong Wang and Zongyang Qiu and Qixun Wang and Zhuoran Zhang and Xuanyu Zhu and Yifan Xu and Yiyan Ji and Shujie Li and Shixuan Liu and Haotian Wang and Yuanxing Zhang and Pengfei Wan and Chengwei Qin},
  title = {{EgoCaptioner: Towards Comprehensive and Fine-Grained Egocentric Video Captioning}},
  year = {2026},
  eprint = {2611.26380},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26380},
}
Figure