Introduction
Dense Video Object Captioning (DVOC) represents a significant challenge in computer vision, requiring models to simultaneously detect, track, and describe the behavior of every entity within a scene. While progress has been made in individual tasks like object detection and tracking, integrating these with natural language descriptions has been hindered by the "annotation bottleneck." Manually labeling videos with precise masks and descriptive captions is prohibitively expensive, leading previous research to rely on disjointed training pipelines that lack a unified spatial and linguistic understanding.
Methodology
Synthetic Data Generation
To overcome data scarcity, the authors developed a sophisticated synthetic data generation pipeline leveraging Gemini 2.0 Flash. Instead of simple video-to-text generation, they employed a multi-modal prompting strategy. By providing the Vision Language Model (VLM) with bounding box coordinates, object categories, and relative areas, the model is forced to focus on specific object trajectories. This spatial grounding significantly improves caption quality, raising human preference ratings from 26.8 to 85.1 compared to "blind" captioning.
End-to-End Architecture
MaskCaptioner is an end-to-end model that bridges localization and language. It builds upon OVFormer, an open-vocabulary video instance segmenter, by adding a BLIP-2 style captioning head. The model processes video clips and aggregates temporal features to generate coherent descriptions of object movements over time. Unlike disjoint methods, MaskCaptioner is trained on unified (mask, box, category, caption) quadruplets, allowing it to learn the direct relationship between visual appearance and action.
Datasets and Evaluation
The research introduces two new synthetically enhanced datasets: LVISCap (1.2M object-level image captions) and LV-VISCap (16k video trajectories across 1,020 classes). Evaluation was conducted across three major benchmarks:
- VidSTG: For spatio-temporal grounding.
- VLN: Video Localized Narratives.
- BenSMOT: Human-centric tracking and captioning.
Key Results and Insights
MaskCaptioner achieves state-of-the-art performance across all benchmarks. On VidSTG, it reached a 64.0 CHOTA score, a substantial improvement over the previous best of 56.9. Captioning accuracy (CapA) also saw a significant boost from 39.7 to 55.4. On the BenSMOT benchmark, the model nearly doubled the CIDEr score from 25.4 to 42.6, demonstrating superior fidelity in describing human actions.
Furthermore, the study reveals that synthetic data scales logarithmically with captioning accuracy, suggesting that further data expansion will continue to drive performance gains. Efficiency is another major advantage; MaskCaptioner requires only 208 GPU hours for training, compared to over 2,000 hours for previous disjointed state-of-the-art methods.
Conclusion
MaskCaptioner demonstrates that end-to-end joint training, supported by high-quality synthetic data, is a powerful approach for dense video understanding. While challenges remain in handling very small objects and ensuring temporal consistency in highly complex scenes, this work sets a new standard for DVOC and provides a scalable path forward for future research in multi-modal video analysis.
Summary updated