Abstract
Universal video understanding requires modeling fine-grained visual and audio information over time in diverse real-world scenarios. However, the performance of existing models is primarily constrained by video-instruction data that represents complex audiovisual content as single, incomplete descriptions, lacking fine-grained organization and reliable annotation. To address this, we introduce: (i) ASID-1M, an open-source collection of one million structured, fine-grained audiovisual instruction annotations with single- and multi-attribute supervision; (ii) ASID-Verify, a scalable data curation pipeline for annotation, with automatic verification and refinement that enforces semantic and temporal consistency between descriptions and the corresponding audiovisual content; and (iii) ASID-Captioner, a video understanding model trained via Supervised Fine-Tuning (SFT) on the ASID-1M. Experiments across seven benchmarks covering audiovisual captioning, attribute-wise captioning, caption-based QA, and caption-based temporal grounding show that ASID-Captioner improves fine-grained caption quality while reducing hallucinations and improving instruction following. It achieves state-of-the-art performance among open-source models and is competitive with Gemini-3-Pro.
Introduction: The Challenge of Fine-Grained Video Data
To teach an AI to understand video, researchers typically use video-text pairs. However, existing video instruction data is fundamentally flawed. Most datasets rely on a single, monolithic prompt that generates a generic, incomplete caption, often missing critical details like temporal localization or camera movements.
Simply merging outputs from different specialized models (ensembling) increases semantic coverage but often introduces conflicting information and "hallucinations." Because models learn directly from this data, unreliable and unstructured captions create a performance ceiling for fine-grained, controllable video understanding. This paper addresses these limitations through a new dataset, ASID-1M, and a verification-centric curation pipeline.
ASID-Verify: A Multi-Stage Curation Pipeline
The authors propose ASID-Verify, a multi-stage pipeline designed to create high-quality, structured annotations through three key stages:
- Multi-Source Generation: Diverse candidate captions are generated using a combination of audio-visual models, Automatic Speech Recognition (ASR) for speech transcription, and MLLM annotators.
- Ensembling & Verification: A Large Language Model (LLM) merges these candidates into a single draft. It then checks the draft against the original audio transcripts and timestamps for consistency.
- Attribute-Based Evaluation & Refinement: This is the core innovation. The pipeline decomposes the draft caption into specific semantic attributes (e.g., Scene, Character, Action, Camera, Speech). An MLLM independently evaluates each attribute for factual errors, missing details, and fluency, refining only the flawed attributes to prevent errors from cascading.
The ASID-1M Dataset and Training Strategy
Using the ASID-Verify pipeline, the authors curated ASID-1M, an open-source collection of one million structured, fine-grained audiovisual instruction annotations. The raw videos were sampled from the YouTube subset of LLaVA-Video-178K and short-form videos from FineVideo.
Using this data, the authors trained ASID-Captioner (built on the Qwen2.5-Omni architecture) using a Progressive Attribute Learning strategy. The model was trained in three stages: first on single attributes (e.g., "Describe only the camera movement"), then jointly on all attributes for short clips, and finally on all attributes over longer temporal contexts (up to 3 minutes). This curriculum-based approach drastically improves the model's ability to follow complex user prompts.
Experimental Results
ASID-Captioner was tested across seven benchmarks covering audiovisual captioning, QA, and temporal grounding.
- Quality Improvements: The multi-stage refinement process significantly improved annotation quality. As shown in the stage-wise analysis, the "No-Error Rate" and "Clean Rate" increased substantially through the refinement passes, while hallucination and missing rates dropped compared to original ground truth data.
- Hallucination Reduction: On the Video-SALMONN-2 benchmark, ASID-Captioner 7B achieved an error score of 35.9, matching the performance of the proprietary Gemini-3-Pro (35.7) and far exceeding other open-source models like InternVL3.5 (79.4).
- Instruction Following: The model demonstrated a massive leap in instruction following. When prompted to describe specific attributes, ASID-Captioner 7B succeeded 58.0% of the time, compared to just 2.5% for the base Qwen2.5-Omni model.
Conclusion
ASID-1M and the ASID-Verify pipeline provide a robust framework for scaling high-quality video instruction data. By shifting from monolithic captions to attribute-structured, verified descriptions, ASID-Captioner achieves state-of-the-art performance in fine-grained video understanding. While the current work focuses on short-form video and relies on proprietary models for verification, it sets a new standard for open-source video MLLM performance.
Summary updated