TL;DR
Researchers have developed an automated pipeline to generate millions of training triplets for Composed Video Retrieval (CoVR) by leveraging Large Language Models (LLMs) to describe differences between mined video pairs. By training on these massive, automatically constructed datasets, their model achieves state-of-the-art zero-shot performance on standard image retrieval benchmarks and establishes a new high-quality benchmark for video-based search.
What Problem Does This Paper Address?
Imagine you have a photo of a fountain during the day, but you want to find a video of that same fountain at night with a fireworks show. This is Composed Video Retrieval (CoVR): searching a database using a combination of a visual query (an image or video) and a text query that specifies a modification.
While powerful, this sub-field of computer vision is severely bottlenecked by data. Training these models requires "triplets": a source image, a text instruction (e.g., "add fireworks"), and a target video. Manually annotating these is incredibly expensive and slow, resulting in existing datasets that are too small to train robust, large-scale models. Furthermore, most research has focused on static images (CoIR), leaving the dynamic, temporal nature of video largely unexplored.
Methods
The authors propose a scalable, fully automatic methodology to generate training data from existing large-scale video-caption datasets (like WebVid2M).
- Mining Pairs: They search for pairs of videos whose captions differ by exactly one word (e.g., "Young man working with tablet..." vs. "Young man working with smartphone..."). The intuition is that similar captions likely correspond to visually similar videos with a specific difference.
- MTG-LLM: To turn these caption differences into natural language instructions, they fine-tune a LLaMA-7B model (dubbed the Modification Text Generation LLM) on a small set of 715 manual annotations. This model takes two captions as input and outputs a concise modification text like "change tablet for smartphone."
- Model Architecture: Their retrieval model, CoVR-BLIP-2, builds on the BLIP-2 architecture. It uses a Querying Transformer (Q-Former) to fuse image and text features.
- Query Scoring for Videos: To handle video targets, the model samples multiple frames. It uses a "query scoring" mechanism to weight these frames based on their relevance to the text modification, allowing the model to focus on the most informative parts of a video clip.
Datasets
The paper introduces several massive datasets and a new evaluation benchmark:
- WebVid-CoVR: 1.6 million video-text-video triplets automatically generated from WebVid2M.
- CC-CoIR: 3.3 million image-text-image triplets generated from the Conceptual Captions dataset.
- WebVid-CoVR-Test: A new, high-quality benchmark consisting of 2,435 manually verified and cleaned triplets to evaluate CoVR performance.
- Existing Benchmarks: The models were also evaluated on standard image-based benchmarks: CIRR, FashionIQ, and CIRCO.
Key Insights and Ideas
- Real Data Trumps Synthetic: A major finding is that training on "real" web-scraped images/videos mined through this pipeline outperforms training on synthetic images generated by diffusion models (like InstructPix2Pix). The diversity and realism of web data provide a better training signal.
- Captions are Richer than Triplets: The authors introduced a composed caption retrieval loss. Since their pipeline provides the actual caption of the target video, they can force the model to align the multi-modal query not just with the target video, but also with its descriptive text. This extra supervision significantly boosts performance.
- Video-to-Image Transfer: Models trained on video data (WebVid-CoVR) transfer remarkably well to image-only tasks. This suggests that learning temporal dynamics and diverse video scenes helps build more robust visual representations for static image retrieval.
Key Results
The proposed approach sets a new bar for zero-shot composed retrieval:
- State-of-the-Art Zero-Shot: On the CIRR benchmark, the model achieved 43.74% R@1, a significant jump over previous competitors.
- CIRCO Benchmark: It reached 28.29 mAP@5, outperforming the nearest competitor by roughly 2%.
- Scaling Laws: Performance steadily improved as the number of training triplets increased from 11k to 1.6M, suggesting that the limit of this automatic construction method hasn't been reached yet.
Limitations and Future Work
Despite the success, the authors note a few key limitations:
- One-Word Delta: The current mining strategy only looks for single-word differences in captions. This prevents the model from learning more complex, multi-attribute modifications (e.g., "a dog swimming with a ball in its mouth").
- Blind Text Generation: The MTG-LLM generates modification text based only on the captions, without looking at the actual video frames. This can lead to "noisy" instructions if the captions don't perfectly capture the visual differences.
- Future Directions: The authors suggest that future work should incorporate "visually-grounded" modification generation, where the LLM sees the images/videos to describe their differences more accurately.
Summary updated