Abstract
Current multimodal learning paradigms, such as CLIP, often rely on a "one-to-one" alignment assumption where an image is paired with a single ground-truth caption. This paper argues that this assumption is fundamentally flawed due to "multiplicity"—the inherent many-to-many relationship between modalities. Ignoring this leads to training instabilities, the "modality gap," and unreliable evaluation metrics. We formalize multiplicity and demonstrate its impact on dataset construction, model training, and performance measurement.
The Challenge of Multiplicity
In large-scale multimodal models, we typically assume that for every image, there is one "ground truth" caption. However, a single image can be described in infinite ways, and a single caption can describe thousands of different images. This phenomenon is termed multiplicity.
Our current machine learning pipelines treat any pair not explicitly labeled as "positive" as a "negative." Because it is impossible to annotate every possible valid combination in a large dataset, models are penalized for finding valid but unannotated matches. This "matching ambiguity" creates a bottleneck in building datasets and measuring success. Unlike unimodal tasks like ImageNet, where labels are fixed, multimodal relations grow quadratically ($O(N^2)$) with dataset size.
Theoretical and Empirical Analysis
The paper provides a mathematical and empirical breakdown of how multiplicity affects learning:
- Gradient Analysis of Contrastive Loss: When a model encounters a "false negative" (a valid match labeled as negative), the resulting gradient mismatch pushes semantically similar items apart. This contributes to the "modality gap" and slows convergence.
- Metric Critique: Standard retrieval metrics like Recall@K (R@K) are misleading. R@K only checks if the single annotated positive is found, ignoring whether other top results are also valid. Analysis of the ECCV Caption benchmark shows that human preference correlates better with ranking-based metrics than with R@K.
Key Insights from Datasets
Multiplicity arises from several factors, including semantic abstraction and representational asymmetry. The paper highlights these through various datasets:
- MS-COCO: Human verification reveals approximately 4.4x more valid matches than original annotations suggest. On average, a caption might match 8.5 images, even if written for only one.
- LAION-5B: Current "alignment-based" filtering (using CLIP scores) might actually be suppressing natural multiplicity to fit the one-to-one paradigm.
Results and Future Directions
The paper demonstrates that mAP@R (mean Average Precision at R) correlates much better with human preference than R@K. Furthermore, the persistent "modality gap" in joint embedding spaces is likely a direct consequence of models trying to satisfy conflicting one-to-one constraints in a many-to-many reality.
Future Research Directions: * Probabilistic Embeddings: Representing images as distributions or sets of vectors to capture multiple concepts. * Multiplicity-Aware Training: Moving toward "soft" labels or positive-unlabeled learning frameworks. * Better Benchmarking: Shifting industry standards toward ranking-based metrics like mAP@R that reward diversity. * Conditional Modeling: Using context or task inputs to resolve ambiguity (e.g., "describe this image focusing on the background").
Summary updated