In multi-modal learning, the standard approach is to align disparate data types—such as images and text—into a shared latent space. Models like CLIP achieve this by maximizing the similarity between matching pairs while pushing non-matching pairs apart. However, this "instance-level" alignment often ignores the internal structure of the individual modalities. When a model focuses solely on bridging the gap between an image and its specific caption, it can inadvertently "fragment" the representation space, losing the semantic relationships that existed within the original data.
Recent research has converged on a solution: intra-modality structure preservation. By ensuring that the relational topology of a single modality remains intact during alignment, models become more robust to noise, generalize better to unseen categories, and maintain a more coherent semantic manifold.
The Fragmentation Problem
Standard contrastive learning treats every non-matching pair as a negative. As Zolfaghari et al. and Liu et al. argue, this is often too rigid. In a large dataset, two different images might both depict "dogs," but a standard contrastive loss might push their embeddings apart simply because they are not the same instance. This neglect of within-view relationships means that semantically similar items from the same domain may not be clustered together in the shared latent space, leading to suboptimal retrieval performance Wang et al..
Kovashka notes that this fragmentation is particularly damaging when images and text are "complementary" (symbolic) rather than "redundant" (literal). Without explicit constraints, the model may "tear apart" the natural manifold of a modality to satisfy the cross-modal alignment objective.
Preserving the Semantic Topology
The core intuition behind structure preservation is that the relative distances between samples in one modality should be mirrored in the other. If "cat" and "kitten" are close in the text embedding space, their corresponding visual representations should also be close in the joint embedding space.
Several mechanisms have been proposed to enforce this:
- Similarity Matrix Alignment: Zhao et al. propose Representation Distribution Alignment (RDA), which aligns the similarity matrices derived from both the feature and label spaces. Similarly, Li et al. use Consistent Structural Relation Learning (CSRL) to ensure that category-level relationships (like the similarity between a cow and a horse) are consistent between word embeddings and generated visual features.
- Neighborhood Consistency: Instead of looking at global matrices, some methods focus on local neighborhoods. Zhao et al. introduce Geometrical Structure Consistency (GSC), which checks if the "neighbors" of an image in the visual space match the "neighbors" of its paired text in the linguistic space. If the neighborhoods diverge, the pair is likely "noisy" or mismatched.
- Intra-Modal Triplet Losses: To prevent the shared space from collapsing, the authors of several papers Wang et al., Liu et al., Sebastian et al. incorporate intra-modal triplet or hinge losses. These force the model to maintain a margin between similar and dissimilar items within the same modality, acting as a powerful regularizer for the cross-modal task.
Why Relational Consistency Works
Relational consistency works because it leverages the most stable signal available: the internal structure of the data itself. While the mapping between modalities can be noisy or sparse, the relationship between two images or two sentences is often much clearer.
- Robustness to Noise: In datasets with "noisy correspondence" (mismatched pairs), direct alignment fails. However, by treating the internal structure of a single modality as a stable reference, systems can identify and correct noisy cross-modality distance measurements Jia et al..
- Generalization and Zero-Shot Learning: For unseen classes, a model cannot rely on learned cross-modal pairs. By preserving the semantic manifold—ensuring that the variance observed in one modality is mirrored in the other—models can ground language to physical objects more effectively Nguyen et al.. Swetha et al. demonstrate that this preservation is a key driver for zero-shot retrieval and out-of-domain generalization.
- Stability in Domain Shift: When moving from synthetic to real-world data, direct alignment often breaks. Goodge et al. show that while domain shifts degrade cross-modal alignment, the relative clustering of similar samples within a single modality remains stable. Prioritizing these internal relationships allows models to maintain performance in messy, real-world environments.
Higher-Order Constraints
Beyond simple similarity, researchers have explored more complex structural constraints. Dou et al. apply cycle-consistency to ensure a one-to-one correspondence between identities in different modalities, ensuring that an identity mapped from RGB to Infrared and back remains consistent. Wang et al. propose "equivariant similarity," requiring that semantic changes (like swapping an image) result in predictable, consistent shifts in the similarity score, regardless of the modality.
By shifting the focus from matching instances to preserving relationships, this research direction provides a framework for building multi-modal models that are not just associative, but structurally coherent.