The style transformer with common knowledge optimization for image-text retrieval

Wenrui Li, Zhengyu Ma, Jinqiao Shi, Xiaopeng Fan

Abstract

Image-text retrieval, the task of associating multi-modal data, is essential for modern search and recommendation systems. However, existing methods often fail to account for high-level semantic relationships—referred to as "style embeddings"—and the shared common knowledge between modalities. We introduce the Common Knowledge Style Transformer Network (CKSTN), a novel framework designed to bridge this gap. CKSTN utilizes a Common Knowledge Adaptor (CKA) comprising a Style Embedding Extractor (SEE) and a Common Knowledge Optimization (CKO) module. By employing a sequential update strategy and a lightweight transformer architecture, CKSTN achieves state-of-the-art performance on MSCOCO and Flickr30K benchmarks while maintaining a significantly lower parameter count than competing models.

Introduction

The ability to accurately retrieve images based on textual descriptions (and vice versa) is a cornerstone of cross-modal AI. While recent advancements have improved fine-grained alignment—matching specific words to specific image regions—they often overlook the "big picture." Specifically, two areas remain underserved: high-level semantic "styles" (the overarching context of a scene) and the dynamic common knowledge that links different modalities (e.g., the conceptual link between the word "ball" and a spherical object in an image).

Furthermore, many high-performing retrieval models are computationally expensive, relying on massive architectures that are difficult to deploy in real-time or resource-constrained environments. CKSTN addresses these challenges by prioritizing semantic efficiency and lightweight design.

The CKSTN Framework

The core of the CKSTN architecture is the Common Knowledge Adaptor (CKA), which integrates two specialized modules to refine feature representation.

Style Embedding Extractor (SEE)

Rather than relying solely on raw feature extraction, the SEE identifies high-level semantic relationships. It processes features row-by-row using Multi-Layer Perceptrons (MLPs) and employs a concatenation and shuffle strategy. This ensures that global information is distributed across feature maps, allowing the model to capture both broad conceptual styles and precise pixel-level alignments.

Common Knowledge Optimization (CKO)

The CKO module is designed to dynamically capture latent concepts shared between images and text. Unlike static lookup tables, the CKO uses a "memory gate" to selectively update common feature units based on the input context. This allows the model to "reason" about the relationship between modalities. To ensure generalized temporal knowledge, a sequential update strategy integrates features from different SEE layers with previous common feature units.

Lightweight Transformer Architecture

To maximize practicality, CKSTN replaces standard, heavy attention layers with a lightweight feed-forward network. This design choice significantly reduces the parameter count and computational overhead without sacrificing the model's ability to learn complex representations.

Experimental Results

Performance Benchmarks

CKSTN was evaluated on the MSCOCO and Flickr30K datasets, demonstrating superior performance compared to existing state-of-the-art methods:

  • MSCOCO: The model achieved an 11.8% relative improvement in Recall@1 (R@1) for image retrieval compared to the DERN method.
  • Flickr30K: CKSTN outperformed the LTAN model by 3.1% on image retrieval and 2.4% on sentence retrieval (R@1).

Parameter Efficiency

One of the most significant advantages of CKSTN is its efficiency. The model achieves these results with only 175 million parameters, compared to the 275 million parameters required by models like ViLBERT. This suggests that sophisticated semantic modeling is more effective for retrieval tasks than simply increasing model depth or width.

Discussion and Insights

The "Style" Factor and Stability

By treating high-level semantic relationships as a "style" to be extracted, CKSTN gains a global understanding of scenes. This helps resolve ambiguities that fine-grained matching often misses. Additionally, the CKO module acts as a stabilizer; in many lightweight models, performance drops sharply when intermediate layer dimensions are reduced (the "cliff effect"). CKO optimizes feature representation to prevent this degradation.

Dynamic Reasoning

The success of the sequential update strategy indicates that common knowledge should not be static. By adjusting its internal "common sense" based on the specific image-text pair being analyzed, the model achieves a more nuanced alignment.

Conclusion

CKSTN represents a significant step forward in making image-text retrieval both more accurate and more efficient. By focusing on high-level style embeddings and dynamic common knowledge optimization, the model achieves state-of-the-art results with a lightweight footprint. Future work will focus on testing the model's generalization in specialized domains, such as medical imagery, and further optimizing inference latency for edge device deployment.

Summary updated

Abstract

Image-text retrieval which associates different modalities has drawn broad attention due to its excellent research value and broad real-world application. However, most of the existing methods haven't taken the high-level semantic relationships ("style embedding") and common knowledge from multi-modalities into full consideration. To this end, we introduce a novel style transformer network with common knowledge optimization (CKSTN) for image-text retrieval. The main module is the common knowledge adaptor (CKA) with both the style embedding extractor (SEE) and the common knowledge optimization (CKO) modules. Specifically, the SEE uses the sequential update strategy to effectively connect the features of different stages in SEE. The CKO module is introduced to dynamically capture the latent concepts of common knowledge from different modalities. Besides, to get generalized temporal common knowledge, we propose a sequential update strategy to effectively integrate the features of different layers in SEE with previous common feature units. CKSTN demonstrates the superiorities of the state-of-the-art methods in image-text retrieval on MSCOCO and Flickr30K datasets. Moreover, CKSTN is constructed based on the lightweight transformer which is more convenient and practical for the application of real scenes, due to the better performance and lower parameters.
BibTeX
@article{li2023style,
  author = {Wenrui Li and Zhengyu Ma and Jinqiao Shi and Xiaopeng Fan},
  title = {{The style transformer with common knowledge optimization for image-text retrieval}},
  journal = {IEEE Signal Processing Letters},
  year = {2023},
  url = {https://openreview.net/forum?id=fXoptLgllV},
}
Figure