Structured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrieval

Xuri Ge, Fuhai Chen, Joemon M. Jose, Zhilong Ji, Zhongqin Wu, Xiao Liu

Abstract

Current state-of-the-art image-sentence retrieval methods typically rely on implicit alignment of visual-textual fragments, such as image regions and words, often utilizing attention modules to highlight cross-modal correspondences. However, these methods frequently lack consistent representation in both semantic and structural spaces, leading to suboptimal retrieval performance. This paper introduces the Structured Multi-modal Feature Embedding and Alignment (SMFEA) model. SMFEA addresses these limitations by (i) constructing intrinsic structures among fragments within each modality and (ii) seeking explicit inter-modal structural and semantic correspondence. By utilizing a novel multi-modal structured module with a shared context-aware referral tree, SMFEA explicitly aligns heterogeneous image-sentence data. Extensive experiments on Microsoft COCO and Flickr30K benchmarks demonstrate that SMFEA significantly outperforms existing state-of-the-art methods.

Introduction

Image-sentence retrieval is a fundamental task in cross-modal AI, requiring the ability to find the most relevant image for a given text query and vice versa. While deep learning has advanced this field significantly, bridging the "semantic gap" between visual pixels and linguistic tokens remains a challenge. Most contemporary models treat images as a "bag-of-regions" and sentences as a "bag-of-words," using soft attention to find matches. The Structured Multi-modal Feature Embedding and Alignment (SMFEA) model represents a shift toward structural awareness, ensuring that both modalities are represented through a consistent, hierarchical framework.

The Challenge: Beyond Fragment Matching

Traditional and recent retrieval methods often fail to capture the nuanced relationships between entities in a scene. For instance, a model might correctly identify "dog" and "ball" but fail to distinguish between "a dog chasing a ball" and "a ball hitting a dog."

Comparison of (a) traditional methods, (b) recent retrieval methods, and (c) the proposed SMFEA method.

As shown in Figure 1, while recent methods (b) attempt to align fragments, they often do so implicitly. This lack of explicit structural modeling means the model does not truly "understand" the scene's composition. SMFEA (c) addresses this by building intra-modal trees that are explicitly aligned through inter-modal consistency.

The SMFEA Framework

The core of SMFEA is the creation of a shared structural space for both images and text. The architecture is divided into three main stages: feature extraction, structured embedding, and multi-modal alignment.

The overall architecture of the SMFEA model, featuring visual and textual tree encoders aligned via a shared referral tree.

Context-Aware Tree Encoders

Instead of flat feature vectors, SMFEA constructs three-layer binary trees for every image and sentence.

  • VCS-Tree (Visual Context-aware Structured Tree): Processes image regions extracted via Faster R-CNN. It uses a specialized LSTM-based ternary tree encoder to model how objects (leaf nodes) combine into higher-level actions and relations (parent nodes).
  • TCS-Tree (Textual Context-aware Structured Tree): Similarly processes sentence fragments, capturing the linguistic hierarchy of the text.

Shared Referral Tree and Alignment

To ensure that the visual and textual trees are semantically comparable, the model employs a Shared Referral Tree. This tree is generated using a linguistic parser (e.g., Stanford Parser) to provide a "ground truth" structure. Both the VCS-Tree and TCS-Tree are forced to categorize their nodes into the same semantic labels (nouns, verbs, prepositions) defined by this referral tree.

Alignment is achieved by minimizing the Kullback-Leibler (KL) divergence between the probability distributions of corresponding nodes in the visual and textual trees. This explicit alignment forces the visual model to learn a structural logic that mirrors the semantic hierarchy of language. Additionally, a Consensus-Aware Concept (CAC) module integrates external commonsense knowledge to refine representations of ambiguous concepts.

Experimental Evaluation

The SMFEA model was rigorously tested on the Flickr30K and MS-COCO datasets, which are standard benchmarks for cross-modal retrieval.

Performance Metrics

SMFEA achieved state-of-the-art (SOTA) results across nearly all metrics:

  • Flickr30K: The model reached an rSum (sum of Recall@1, 5, and 10 for both directions) of 487.5, surpassing previous leaders like CVSE and CAAN.
  • MS-COCO (5K split): On this larger and more challenging benchmark, SMFEA achieved a Recall@1 of 54.2% for sentence retrieval and 41.9% for image retrieval.

Robustness and Generalization

A key finding was the model's ability to generalize. When trained on MS-COCO and tested on Flickr30K without fine-tuning, SMFEA maintained high performance. This suggests that the model learns robust, universal semantic structures rather than just dataset-specific patterns.

Discussion and Conclusion

The primary insight of SMFEA is that structure serves as the bridge between modalities. While pixels and words are fundamentally different, the underlying logic of a scene—objects interacting through relations within a global context—is universal. By making these relationships explicit through tree-based modeling and distribution alignment, SMFEA significantly narrows the semantic gap.

Despite its success, the model has limitations. The current fixed three-layer, seven-node tree structure may be too simple for extremely complex scenes with many interacting objects. Future work will explore more dynamic and fine-grained tree structures to capture even greater levels of detail in visual and textual data.

Summary updated

Abstract

The current state-of-the-art image-sentence retrieval methods implicitly align the visual-textual fragments, like regions in images and words in sentences, and adopt attention modules to highlight the relevance of cross-modal semantic correspondences. However, the retrieval performance remains unsatisfactory due to a lack of consistent representation in both semantics and structural spaces. In this work, we propose to address the above issue from two aspects: (i) constructing intrinsic structure (along with relations) among the fragments of respective modalities, e.g., "dog $\to$ play $\to$ ball" in semantic structure for an image, and (ii) seeking explicit inter-modal structural and semantic correspondence between the visual and textual modalities. In this paper, we propose a novel Structured Multi-modal Feature Embedding and Alignment (SMFEA) model for image-sentence retrieval. In order to jointly and explicitly learn the visual-textual embedding and the cross-modal alignment, SMFEA creates a novel multi-modal structured module with a shared context-aware referral tree. In particular, the relations of the visual and textual fragments are modeled by constructing Visual Context-aware Structured Tree encoder (VCS-Tree) and Textual Context-aware Structured Tree encoder (TCS-Tree) with shared labels, from which visual and textual features can be jointly learned and optimized. We utilize the multi-modal tree structure to explicitly align the heterogeneous image-sentence data by maximizing the semantic and structural similarity between corresponding inter-modal tree nodes. Extensive experiments on Microsoft COCO and Flickr30K benchmarks demonstrate the superiority of the proposed model in comparison to the state-of-the-art methods.
BibTeX
@inproceedings{ge2021structured,
  author = {Xuri Ge and Fuhai Chen and Joemon M. Jose and Zhilong Ji and Zhongqin Wu and Xiao Liu},
  title = {{Structured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrieval}},
  booktitle = {ACM Multimedia},
  year = {2021},
  url = {https://openreview.net/forum?id=sbJ8OFY4tO},
}
Figure