Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning

Sanghyuk ChunarXiv 2025 · DOI · arXiv

Abstract

Current multimodal learning paradigms, such as CLIP, often rely on a "one-to-one" alignment assumption where an image is paired with a single ground-truth caption. This paper argues that this assumption is fundamentally flawed due to "multiplicity"—the inherent many-to-many relationship between modalities. Ignoring this leads to training instabilities, the "modality gap," and unreliable evaluation metrics. We formalize multiplicity and demonstrate its impact on dataset construction, model training, and performance measurement.

The Challenge of Multiplicity

In large-scale multimodal models, we typically assume that for every image, there is one "ground truth" caption. However, a single image can be described in infinite ways, and a single caption can describe thousands of different images. This phenomenon is termed multiplicity.

Comparison of unimodal (many-to-one) and multimodal (many-to-many) tasks, illustrating how multiplicity creates complex correspondences.

Our current machine learning pipelines treat any pair not explicitly labeled as "positive" as a "negative." Because it is impossible to annotate every possible valid combination in a large dataset, models are penalized for finding valid but unannotated matches. This "matching ambiguity" creates a bottleneck in building datasets and measuring success. Unlike unimodal tasks like ImageNet, where labels are fixed, multimodal relations grow quadratically ($O(N^2)$) with dataset size.

Theoretical and Empirical Analysis

The paper provides a mathematical and empirical breakdown of how multiplicity affects learning:

  • Gradient Analysis of Contrastive Loss: When a model encounters a "false negative" (a valid match labeled as negative), the resulting gradient mismatch pushes semantically similar items apart. This contributes to the "modality gap" and slows convergence.
  • Metric Critique: Standard retrieval metrics like Recall@K (R@K) are misleading. R@K only checks if the single annotated positive is found, ignoring whether other top results are also valid. Analysis of the ECCV Caption benchmark shows that human preference correlates better with ranking-based metrics than with R@K.

Key Insights from Datasets

Multiplicity arises from several factors, including semantic abstraction and representational asymmetry. The paper highlights these through various datasets:

  • MS-COCO: Human verification reveals approximately 4.4x more valid matches than original annotations suggest. On average, a caption might match 8.5 images, even if written for only one.
  • LAION-5B: Current "alignment-based" filtering (using CLIP scores) might actually be suppressing natural multiplicity to fit the one-to-one paradigm.

Results and Future Directions

The paper demonstrates that mAP@R (mean Average Precision at R) correlates much better with human preference than R@K. Furthermore, the persistent "modality gap" in joint embedding spaces is likely a direct consequence of models trying to satisfy conflicting one-to-one constraints in a many-to-many reality.

Future Research Directions: * Probabilistic Embeddings: Representing images as distributions or sets of vectors to capture multiple concepts. * Multiplicity-Aware Training: Moving toward "soft" labels or positive-unlabeled learning frameworks. * Better Benchmarking: Shifting industry standards toward ranking-based metrics like mAP@R that reward diversity. * Conditional Modeling: Using context or task inputs to resolve ambiguity (e.g., "describe this image focusing on the background").

Summary updated

Abstract

Multimodal learning has seen remarkable progress, particularly with large-scale pre-training across various modalities. Most current approaches are built on the assumption of a deterministic one-to-one alignment between modalities. However, this oversimplifies real-world multimodal relationships, where their nature is inherently many-to-many. The many-to-many property, or multiplicity, is not a side-effect of noise or annotation error, but an inevitable outcome of intra-modal variability, representational asymmetry, and task-dependent ambiguity in multimodal tasks. We argue that multiplicity is a fundamental bottleneck that affects all stages of the multimodal learning pipeline: from data construction to model training and evaluation benchmarks. By formalizing its causes and consequences, we demonstrate how ignoring multiplicity leads to training uncertainty, unreliable evaluation, and degraded dataset quality. This position paper calls for new research directions on multimodal learning, including multiplicity-aware learning frameworks and dataset construction and evaluation protocols.
BibTeX
@misc{chun2025multiplicity,
  author = {Sanghyuk Chun},
  title = {{Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning}},
  year = {2025},
  eprint = {2505.19614},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.19614},
}
Figure