Abstract
Cross-modal retrieval—the task of searching for data in one modality (e.g., images) using a query from another (e.g., text)—is fundamentally challenged by the "heterogeneity gap." This paper introduces Wasserstein Coupled Graph Learning (WCGL), a framework that bridges this gap by representing data as graphs and mapping them into a shared dictionary space. By utilizing the Wasserstein distance for structural comparison, WCGL achieves state-of-the-art performance across multiple benchmarks, demonstrating the power of relational structure in cross-modal alignment.
Introduction
Traditional cross-modal retrieval methods often treat images and sentences as "flat" feature vectors, effectively ignoring the rich internal structures and relationships between entities, such as a "person" "holding" a "surfboard." While graph-based methods have been proposed to capture these relationships, they often suffer from high computational complexity or poor alignment between disparate modalities. WCGL addresses these issues by combining Graph Convolutional Networks (GCNs) with an optimal transport-based embedding strategy, allowing for a robust and differentiable comparison of structural information.
The WCGL Framework
The WCGL architecture is built upon three core pillars designed to extract and align structural features:
- Graph Encoding: Input data is transformed into graphs where nodes represent objects or words and edges represent their relationships. These graphs are processed through GCNs to extract high-level structural features.
- Wasserstein Graph Embedding (WGE): The model maintains a Coupled Graph Dictionary—a set of learnable "template" graphs for both vision and text. The similarity between an input graph and these templates is calculated using the entropy-smoothed Wasserstein distance, solved via Sinkhorn iterations. This projects different modalities into a common "dictionary space."
- Discriminant Learning: To ensure semantic alignment, a Wasserstein Discriminant Loss (WD-loss) forces "counterpart" keys (e.g., the visual and textual templates for the same concept) to be close in the embedding space while pushing unrelated templates apart.
Experimental Evaluation
The model was evaluated on four major benchmarks: Flickr30K, MSCOCO, Real-world Scene Graph (RSG), and Moviegraphs. WCGL consistently outperformed existing state-of-the-art (SOTA) methods:
- Flickr30K: Achieved a Recall@1 (R@1) of 74.8% for image-to-text retrieval, surpassing the previous best of 72.6%.
- Moviegraphs: On this highly structural video-to-text dataset, WCGL reached an R@1 of 9.20%, a significant improvement over the previous SOTA (7.40%).
Performance is sensitive to architectural choices such as the number of layers in the graph encoder and the size of the coupled dictionary. Ablation studies confirmed that the shared template space provided by the dictionary is essential for effective cross-modal alignment.
Key Insights
- Structure as a Universal Language: By focusing on relationships rather than just isolated entities, the model finds a more stable bridge between vision and language.
- Soft Matching via Optimal Transport: Using the Wasserstein distance allows for a "soft," differentiable alignment of nodes between graphs, making structural comparison compatible with end-to-end deep learning.
- Dictionary Bottleneck: Comparing inputs to a small set of optimized "concepts" (the dictionary) filters out noise and focuses the model on core semantic meanings.
Limitations and Future Work
Despite its success, WCGL's performance is dependent on the quality of initial graph construction (e.g., object detection accuracy). Additionally, while Sinkhorn iterations make Wasserstein distance feasible, they still introduce more computational overhead than simple dot-product similarities. Future research will explore dynamic graph construction methods and further optimization of the dictionary scaling process.
Summary updated