MaskCaptioner: Learning to Jointly Segment and Caption Object Trajectories in Videos

Gabriel Fiastre, Antoine Yang, Cordelia Schmid

Introduction

Dense Video Object Captioning (DVOC) represents a significant challenge in computer vision, requiring models to simultaneously detect, track, and describe the behavior of every entity within a scene. While progress has been made in individual tasks like object detection and tracking, integrating these with natural language descriptions has been hindered by the "annotation bottleneck." Manually labeling videos with precise masks and descriptive captions is prohibitively expensive, leading previous research to rely on disjointed training pipelines that lack a unified spatial and linguistic understanding.

Methodology

Synthetic Data Generation

To overcome data scarcity, the authors developed a sophisticated synthetic data generation pipeline leveraging Gemini 2.0 Flash. Instead of simple video-to-text generation, they employed a multi-modal prompting strategy. By providing the Vision Language Model (VLM) with bounding box coordinates, object categories, and relative areas, the model is forced to focus on specific object trajectories. This spatial grounding significantly improves caption quality, raising human preference ratings from 26.8 to 85.1 compared to "blind" captioning.

The synthetic data generation pipeline uses Gemini with multi-modal prompts including bounding boxes and metadata to produce high-quality object-level captions.

End-to-End Architecture

MaskCaptioner is an end-to-end model that bridges localization and language. It builds upon OVFormer, an open-vocabulary video instance segmenter, by adding a BLIP-2 style captioning head. The model processes video clips and aggregates temporal features to generate coherent descriptions of object movements over time. Unlike disjoint methods, MaskCaptioner is trained on unified (mask, box, category, caption) quadruplets, allowing it to learn the direct relationship between visual appearance and action.

The MaskCaptioner architecture integrates Mask2former-based instance tracking with a BLIP-2 LLM decoder for unified video-level object captioning.

Datasets and Evaluation

The research introduces two new synthetically enhanced datasets: LVISCap (1.2M object-level image captions) and LV-VISCap (16k video trajectories across 1,020 classes). Evaluation was conducted across three major benchmarks:

  • VidSTG: For spatio-temporal grounding.
  • VLN: Video Localized Narratives.
  • BenSMOT: Human-centric tracking and captioning.

Key Results and Insights

MaskCaptioner achieves state-of-the-art performance across all benchmarks. On VidSTG, it reached a 64.0 CHOTA score, a substantial improvement over the previous best of 56.9. Captioning accuracy (CapA) also saw a significant boost from 39.7 to 55.4. On the BenSMOT benchmark, the model nearly doubled the CIDEr score from 25.4 to 42.6, demonstrating superior fidelity in describing human actions.

Furthermore, the study reveals that synthetic data scales logarithmically with captioning accuracy, suggesting that further data expansion will continue to drive performance gains. Efficiency is another major advantage; MaskCaptioner requires only 208 GPU hours for training, compared to over 2,000 hours for previous disjointed state-of-the-art methods.

Conclusion

MaskCaptioner demonstrates that end-to-end joint training, supported by high-quality synthetic data, is a powerful approach for dense video understanding. While challenges remain in handling very small objects and ensuring temporal consistency in highly complex scenes, this work sets a new standard for DVOC and provides a scalable path forward for future research in multi-modal video analysis.

Summary updated

BibTeX
@inproceedings{fiastre2026maskcaptioner,
  author = {Gabriel Fiastre and Antoine Yang and Cordelia Schmid},
  title = {{MaskCaptioner: Learning to Jointly Segment and Caption Object Trajectories in Videos}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=4xt72x45TO},
}
Figure