In the world of large-scale vision-language pretraining, data is king. But not all data is created equal. Image-text corpora are often riddled with near-duplicates, mismatched captions, and redundant records. Standard deduplication tools usually treat this as a binary problem: either two records are the same, or they aren't.
A new paper, "VL-Dedup: Graded Multimodal Entity Resolution for Image-Text Data Deduplication," argues that this binary view is too simplistic. By introducing "graded" entity resolution, the authors show how we can clean our datasets more effectively without throwing away valuable information.
TL;DR
VL-Dedup is a system for "graded" multimodal deduplication that distinguishes between visual copies, complementary captions, and mismatches. It uses a novel cluster-size cap to prevent massive false merges and leverages Vision-Language Models (VLMs) within clusters to preserve unique facts while purging errors.
What Problem Does This Paper Address?
Building high-quality image-text datasets (like LAION or DataComp) is hard. Current curation pipelines face several distinct challenges:
- Visual Copies: The same image appearing multiple times.
- Complementary Captions: Different captions for the same image that describe different aspects (e.g., one describes the "birds" and another the "church tower").
- Mismatches: Captions that simply don't match the image.
Standard deduplication often uses a "keep-one" policy: if images are similar, keep only one record. This accidentally deletes "complementary" facts. Furthermore, most systems are evaluated on labeled pairs, but in production, these pairs are chained into clusters. A single false-positive edge can chain together thousands of unrelated images into one giant "duplicate" cluster—a phenomenon the authors call "graph errors."
Methods
The authors propose VL-Dedup, a multi-stage pipeline that separates visual identity from caption handling:
- Candidate Retrieval: Uses perceptual hashes (pHash) and CLIP embeddings to find potential matches quickly.
- Pair Scoring & Clustering: A classifier scores pairs, and a Union-Find algorithm with a size cap $K$ groups them. This cap is a critical innovation; it prevents a few wrong edges from merging massive, unrelated parts of the dataset.
- VLM-in-the-Loop: Instead of calling an expensive VLM for every possible pair, VL-Dedup only uses the VLM inside the established clusters. It asks the VLM:
- Does this caption actually match this image? (Mismatch detection)
- Do these two captions provide different, useful facts? (Complementary detection)
- Graded Export: Based on VLM verdicts, the system decides whether to keep both records, delete one as redundant, or flag a record for human review.
Datasets
The authors introduced VL-DedupBench, a benchmark designed to evaluate deduplication at three levels: pairs, clusters, and the final "retained set." * Records: 50,000 records from COCO and Stanford Online Products (SOP). * Labels: 66,788 labeled pairs. * Test Surfaces: To test robustness, they created "edited" versions of images (crops, rotations, color shifts) and included multiple distinct photos of the same retail products.
Key Insights and Ideas
- Labeled Pairs Hide Graph Errors: This is perhaps the most striking finding. A classifier can achieve near-perfect F1 scores on a test set of labeled pairs but still fail miserably in production. Why? Because the few errors it does make can act as bridges, merging millions of unrelated records into a single cluster.
- The Power of the Size Cap: By simply limiting the maximum size of a cluster (e.g., $K=5$), the authors reduced false joins from over 7.7 million down to just 385. This "regularizes" the clustering process against noisy classifier outputs.
- VLMs are Better at Single-Record Judging: The authors found that VLMs are significantly more accurate at identifying mismatches when looking at one image-caption pair at a time rather than comparing two pairs.
Key Results
- Information Preservation: In an audit of 50 test clusters, a standard "keep-one-per-cluster" policy lost between 70 and 141 unique facts. VL-Dedup's graded policy lost only 5 to 15 facts.
- Robustness: On images with "Strong" edits (heavy cropping and rotation), standard copy detectors failed (F1 dropped to 0), while VL-Dedup maintained high performance (F1 ~0.99) thanks to its multimodal features.
- Efficiency: By only calling the VLM inside clusters, the system remains scalable. On a 50k record dataset, the VLM only needed to process ~13% of the pairs that shared a cluster.
Limitations and Future Work
- VLM Cost: While mitigated by clustering, VLM calls are still the primary bottleneck for massive web-scale corpora.
- Threshold Sensitivity: The system relies on several thresholds (for pair scoring and VLM confidence) that need to be tuned on validation data.
- Future Directions: The authors suggest extending this "graded" approach to other curation tasks, such as deduplicating interleaved image-text documents (like blog posts or news articles) where the context is even more complex.
Summary updated