Structured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrieval

Xuri Ge, Fuhai Chen, Joemon M. Jose, Zhilong Ji, Zhongqin Wu, Xiao Liu

TL;DR

The Structured Multi-modal Feature Embedding and Alignment (SMFEA) model improves image-sentence retrieval by moving beyond simple "bag-of-words" or "bag-of-regions" matching. It constructs explicit, hierarchical tree structures for both images and sentences, then aligns these trees using a shared semantic framework to ensure that both modalities "see" the world through the same structural lens.

What Problem Does This Paper Address?

In image-sentence retrieval, the goal is to find the most relevant image for a given text query (or vice versa). Current state-of-the-art methods typically use attention mechanisms to find "fragments"—matching specific words (like "dog") to specific image regions.

However, the authors identify two major flaws in this approach:

  1. Ignoring Context: They often ignore the relationships between fragments (e.g., the difference between "dog chases ball" and "ball hits dog").
  2. Implicit Alignment: They rely on "soft" attention to guess correspondences, which often fails to bridge the "semantic gap" because the visual and textual features are represented in fundamentally different structural spaces.

Methods

The authors propose SMFEA, which introduces several novel components:

  • Context-Aware Tree Encoders (VCS-Tree & TCS-Tree): Instead of flat vectors, the model builds a three-layer binary tree for every image and sentence. The VCS-Tree (Visual) and TCS-Tree (Textual) use a specialized LSTM-based ternary tree encoder to capture how objects (leaf nodes) combine into actions and relations (parent nodes).
  • Shared Referral Tree: To ensure the trees are comparable, the model uses a "referral tree" generated by linguistic parsing (Stanford Parser). This acts as a supervisor, forcing both the visual and textual branches to categorize their nodes into the same semantic labels (e.g., nouns, verbs, prepositions).
  • Multi-Modal Alignment via KL-Divergence: To narrow the gap between modalities, the model minimizes the Kullback-Leibler (KL) divergence between the probability distributions of corresponding nodes in the visual and textual trees. This forces the visual model to learn a structural logic that mirrors the language.
  • Consensus-Aware Concept (CAC) Module: It integrates external commonsense knowledge to refine the semantic representation of concepts that might be ambiguous.
  • Hybrid Objective Function: The model is trained using a combination of bidirectional triplet ranking loss (standard for retrieval), cross-entropy loss (for node classification), and the KL-divergence alignment loss.

Datasets

The model was evaluated on two gold-standard benchmarks for cross-modal retrieval:

  • Flickr30K: Contains 31,000 images, each with five descriptive sentences.
  • MS-COCO: A larger dataset with over 123,000 images. The authors tested on both the 1K and 5K test splits.

Key Insights and Ideas

  • Structure is the Bridge: The core "aha moment" is that while pixels and words look different, the underlying structure of a scene (Objects $\rightarrow$ Relations $\rightarrow$ Global Context) is universal. By enforcing this structure on the visual features, the model makes them much easier to map to language.
  • Explicit vs. Implicit: Most models hope the neural network "figures out" relationships implicitly. SMFEA makes these relationships explicit by using a fixed-structure tree (7 nodes) that represents the core semantic content of a pair.
  • Robustness through Distribution Alignment: Using KL-divergence to align the distributions of node categories (rather than just the raw features) provides better fault tolerance against noisy image regions or irrelevant words.

Key Results

SMFEA achieved state-of-the-art (SOTA) results across nearly all metrics:

  • Flickr30K: Achieved an rSum (the sum of Recall@1, 5, and 10 for both image and text retrieval) of 487.5, outperforming previous SOTA models like CVSE (482.4) and CAAN (478.6).
  • MS-COCO (5K split): On the most challenging benchmark, it reached a Recall@1 of 54.2% for sentence retrieval and 41.9% for image retrieval, showing a significant lead over existing ensemble models.
  • Cross-Dataset Generalization: When trained on MS-COCO and tested on Flickr30K (without fine-tuning), SMFEA maintained high performance, proving it learns robust semantic concepts rather than just memorizing a specific dataset.

Limitations and Future Work

  • Fixed Tree Complexity: The current model uses a fixed three-layer, seven-node tree. While this captures "coarse-grained" semantics (e.g., a girl, a bar, a shirt), it may struggle with highly complex scenes containing dozens of interacting objects.
  • Vocabulary Constraints: The authors note that the model sometimes fails to distinguish between very similar sets (e.g., different types of green dresses).
  • Future Directions: The authors suggest expanding the "referral tree" to include more fine-grained categories and improving the accuracy of the initial fragment/relation extraction to handle more nuanced descriptions.

Summary updated

Abstract

The current state-of-the-art image-sentence retrieval methods implicitly align the visual-textual fragments, like regions in images and words in sentences, and adopt attention modules to highlight the relevance of cross-modal semantic correspondences. However, the retrieval performance remains unsatisfactory due to a lack of consistent representation in both semantics and structural spaces. In this work, we propose to address the above issue from two aspects: (i) constructing intrinsic structure (along with relations) among the fragments of respective modalities, e.g., "dog → play → ball" in semantic structure for an image, and (ii) seeking explicit inter-modal structural and semantic correspondence between the visual and textual modalities. In this paper, we propose a novel Structured Multi-modal Feature Embedding and Alignment (SMFEA) model for image-sentence retrieval. In order to jointly and explicitly learn the visual-textual embedding and the cross-modal alignment, SMFEA creates a novel multi-modal structured module with a shared context-aware referral tree. In particular, the relations of the visual and textual fragments are modeled by constructing Visual Context-aware Structured Tree encoder (VCS-Tree) and Textual Context-aware Structured Tree encoder (TCS-Tree) with shared labels, from which visual and textual features can be jointly learned and optimized. We utilize the multi-modal tree structure to explicitly align the heterogeneous image-sentence data by maximizing the semantic and structural similarity between corresponding inter-modal tree nodes. Extensive experiments on Microsoft COCO and Flickr30K benchmarks demonstrate the superiority of the proposed model in comparison to the state-of-the-art methods.
BibTeX
@inproceedings{ge2021structured,
  author = {Xuri Ge and Fuhai Chen and Joemon M. Jose and Zhilong Ji and Zhongqin Wu and Xiao Liu},
  title = {{Structured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrieval}},
  booktitle = {ACM Multimedia},
  year = {2021},
  url = {https://openreview.net/forum?id=sbJ8OFY4tO},
}

Loading related content...

Figure