TL;DR
This paper introduces VC-Agent, an interactive MLLM-based agent that significantly speeds up the collection of customized video datasets. By iteratively refining user requirements through a conversational interface and novel filtering policies, VC-Agent automates the process, reducing manual effort from weeks to minutes.
What Problem Does This Paper Address?
The explosion of video data has been a boon for AI research, but collecting specialized datasets that meet precise needs remains a major bottleneck. Manually sifting through vast amounts of online video is incredibly time-consuming and labor-intensive. Existing methods struggle to handle complex, evolving user requirements, making it difficult to scale up personalized data collection efficiently.
Methods
VC-Agent tackles this by combining a user-friendly interface with intelligent agent functions.
- User Interface: The agent supports a three-step interaction:
- Initial Query: Users provide a rough textual description of their desired dataset.
- Confirmation: The agent presents candidate videos, and users select which ones they like or dislike.
- Comment: For disliked videos, users provide specific reasons for rejection.
- Agent Functions:
- Video Proposal: Leverages MLLMs and video grounding models to retrieve relevant video clips based on the user's evolving demands.
- Filtering Policy: This is the core innovation. VC-Agent employs two novel, self-updatable policies:
- Template-Based Acceptance Policy: Summarizes positively confirmed videos into "criterion templates" to guide future selections.
- Attribute-Aware Rejection Policy: Analyzes user rejection comments to build a "negative standard table" of undesirable attributes, preventing similar videos from being selected.
- Iterative Refinement: The interaction loop (confirmation/comment) allows both the user's understanding of their needs and the agent's filtering policy to progressively improve.
- Double-Check Strategy: For low-confidence filtering decisions, the agent prompts users for a second look, further refining the policies.
- Automated Scale-Up: Once the user is satisfied, the agent transitions to a fully automatic mode to collect the dataset at scale.
Datasets
The paper introduces a Personalized Video Collection Benchmark (PVB). This benchmark consists of 10,000 internet videos annotated across three distinct domains, each with five specific requirements (e.g., for bird videos: category, action, shot type, obscuration, and content). This differs from traditional video retrieval benchmarks by focusing on iterative, multi-faceted user requirements rather than single-query relevance.
Key Insights and Ideas
- Iterative Refinement is Key: The most significant insight is that mimicking human dataset collection—where requirements are clarified through interaction—is crucial for handling complex, nuanced user needs.
- Dual Policies for Nuance: Separating acceptance (template-based) and rejection (attribute-aware) policies effectively captures the different ways users express preferences and dislikes.
- MLLMs as the "Brain": The success hinges on MLLMs' ability to understand user intent, process multimodal information, and generate descriptive summaries for policy updates.
- Efficiency Gains are Substantial: The agent dramatically reduces the time required for dataset collection, shifting from weeks of manual work to mere minutes of user interaction.
Key Results
- VC-Agent significantly outperforms traditional video retrieval methods and other MLLMs on the PVB, especially as the number of requirements increases. For instance, with all five requirements met, VC-Agent achieves a higher IoU compared to baselines.
- User studies show high satisfaction with the collected data quality and a dramatic reduction in collection time. Participants reported spending an average of 690 hours per hour of collected data, a significant improvement over estimated manual collection times.
- The agent typically requires only about 10 interaction rounds, often including 1-3 double-check rounds, to construct a high-quality dataset.
- Fine-tuning downstream models (text-to-video generation, pose estimation) with data collected by VC-Agent led to better performance and more realistic results compared to data collected using other methods.
Limitations and Future Work
- Complex Requirements: The authors acknowledge that the agent still faces challenges with highly complex or intricate user requirements, particularly those involving subtle temporal aspects like cut detection or motion degree.
- Efficiency and Accuracy: While significantly improved, there's still room for enhancing the agent's efficiency and accuracy, especially for more complex tasks.
- Human Judgment: For highly intricate requirements, human confirmation and judgment are still necessary to ensure data quality.
- Future Work: The authors suggest that future improvements will likely depend on advancements in MLLMs themselves, particularly in their ability to parse and interpret increasingly complex user demands. They also aim to further improve the interactive system's efficiency and accuracy.
Summary updated