TL;DR
This paper introduces a novel, scalable method for automatically constructing large-scale datasets for Composed Video Retrieval (CoVR), a task that requires finding videos based on an image query and a text-based modification. By leveraging existing video-caption pairs and a language model, they create the WebVid-CoVR dataset, significantly advancing the state-of-the-art in CoVR and demonstrating strong transferability to Composed Image Retrieval (CoIR).
What Problem Does This Paper Address?
Composed Image Retrieval (CoIR) and its extension, Composed Video Retrieval (CoVR), are challenging tasks that require users to search for visual content using both an image and a text description of a desired modification. While CoIR has seen progress, CoVR is a relatively new area. A major bottleneck for both tasks is the scarcity of large, manually annotated datasets. Creating these "triplets" (query image/video, modification text, target image/video) is time-consuming and expensive, hindering scalability. This paper tackles this by proposing an automatic, scalable approach to generate these crucial training datasets, enabling more robust and generalizable CoVR models.
Methods
The core of this work lies in their automatic data construction pipeline. It involves:
- Mining Paired Captions: Identifying video-caption pairs from large web datasets where the captions differ by a single word. This leverages the assumption that similar captions often correspond to visually similar videos.
- Filtering Caption Pairs: Applying CLIP text similarity to filter out pairs that are too similar (trivial modifications) or too dissimilar (difficult to describe).
- Modification Text Generation (MTG-LLM): Training a large language model (LLM), specifically finetuning LLaMA, to generate the modification text given the two differing captions. This MTG-LLM is trained on a small, manually annotated dataset of caption pairs and their corresponding modification texts.
- CoVR Model Architecture (CoVR-BLIP-2): Adapting the BLIP-2 model, a powerful vision-language pre-trained model. They use its image encoder for the query image and its Q-Former for video frame encoding, aggregating them into a multi-modal query embedding. A key addition is a "caption retrieval loss" that further leverages the available text supervision.
- Training Strategy: Employing a contrastive learning approach with HN-NCE (Hard Negative-Noise Contrastive Estimation) to handle noisy data and upsample hard negatives. They also iterate over distinct target videos in batches, rather than triplets, to maximize the use of negative samples.
Datasets
- WebVid-CoVR: A newly constructed, large-scale dataset for Composed Video Retrieval, containing 1.6 million video-text-video triplets. It's derived from the WebVid2M dataset.
- CC-CoIR: A new, large-scale dataset for Composed Image Retrieval, containing 3.3 million image-text-image triplets. It's derived from the Conceptual Captions dataset.
- WebVid-CoVR-Test: A manually annotated test set for CoVR, comprising 2,500 triplets, created to provide a reliable evaluation benchmark.
- Existing CoIR Datasets: CIRR, FashionIQ, and CIRCO were used for evaluating the transferability of their models to Composed Image Retrieval.
Key Insights and Ideas
- Scalability through Automation: The most significant insight is that automatic data generation, even if inherently noisy, can be highly effective for training complex vision-language tasks like CoVR, overcoming the limitations of manual annotation.
- Caption Similarity as a Proxy for Visual Similarity: The paper demonstrates that mining video pairs based on caption similarity is a viable strategy for generating training data, even without explicit visual similarity filtering.
- Leveraging Text Supervision: The inclusion of an additional caption retrieval loss, beyond the video-text-video triplet objective, significantly improves performance, highlighting the value of available textual information.
- Transferability of CoVR Data to CoIR: Training on their automatically generated CoVR data (WebVid-CoVR) leads to state-of-the-art results on existing CoIR benchmarks, suggesting a strong generalization capability.
- Real Data Dominance: Training on their automatically generated real-world data (WV-CC-CoVIR) outperforms models trained on synthetically generated image datasets, emphasizing the importance of natural data.
Key Results
- WebVid-CoVR Dataset: The creation of a 1.6 million triplet CoVR dataset, significantly larger than any previous composed retrieval dataset.
- State-of-the-Art Performance: Their CoVR-BLIP-2 model, trained on WebVid-CoVR, achieves state-of-the-art results on the newly introduced WebVid-CoVR-Test benchmark.
- Strong CoIR Transfer: The model trained on their datasets achieves state-of-the-art performance in the zero-shot setting on CIRR, FashionIQ, and CIRCO benchmarks.
- Effectiveness of MTG-LLM: Finetuning an LLM for modification text generation significantly outperforms rule-based or prompting-only approaches.
- Visual vs. Text Modalities: Combining both visual and text modalities consistently yields better performance than individual modalities alone.
Limitations and Future Work
- Noisy Data: While effective, the automatically generated data is inherently noisy. The authors acknowledge that about 22% of their test set was discarded due to noise.
- Suboptimal Modification Text: The modification text generation relies on single-word differences in captions, which might not capture complex, multi-word modifications or visually grounded changes.
- Future Work: The authors suggest exploring visually-grounded modification generation and handling multiple modifications between query and target pairs. They also note the potential for using video queries and investigating the dynamic nature of video content more deeply.
Summary updated