Enhanced Modality Transition for Image Captioning

Ziwei Wang, Yadan Luo, Zi HuangarXiv 2021 · arXiv

Introduction

Image captioning requires a sophisticated bridge between visual perception and linguistic expression. While traditional encoder-decoder architectures have made significant strides, they often struggle with the intrinsic "modality gap" between high-level visual features and the complex semantic space of natural language. This paper introduces the Modality Transition Module (MTM), a novel component designed to explicitly map visual vectors into a textual-like subspace. By aligning visual representations with global textual semantics during training, the MTM enables models to generate more precise and contextually rich descriptions, achieving a significant performance boost over existing state-of-the-art methods.

The Modality Gap in Image Captioning

Standard image captioning models typically utilize a CNN (pre-trained on classification tasks like ImageNet) to extract visual features, which are then decoded by an RNN or LSTM. However, this approach assumes a direct interchangeability between visual features and word embeddings that may not exist. Visual encoders are often optimized for a limited set of categories (e.g., 1,000 classes), whereas language models operate on much larger vocabularies (often exceeding 10,000 words).

This discrepancy—the modality gap—can mislead the decoder, resulting in generic captions that overlook specific details or "highlights" in an image. For instance, a model might describe a "red tennis court" simply as a "tennis court" because the visual feature for "red" wasn't effectively translated into the linguistic context of the scene.

The Modality Transition Framework

The core innovation of this work is the Modality Transition Module (MTM), which serves as a dedicated bridge between the vision and language domains. The framework operates in two distinct phases: training and inference.

Comparison between a standard baseline architecture and the proposed Modality Transition framework.

Training Phase

During training, the model utilizes a pre-trained Text Auto-Encoder to compress ground-truth captions into a global textual representation ($u_g$). Simultaneously, the MTM takes averaged visual features from a region detector (such as Faster-RCNN) and projects them through neural network layers to produce a "preliminary textual vector" ($u'_g$).

The MTM is optimized using a Modality Loss ($L_M$), specifically Mean Squared Error (MSE), to align its output ($u'_g$) with the target global textual vector ($u_g$) from the auto-encoder. This process forces the visual features to "speak the language" of the text subspace before the decoding process even begins.

Inference Phase

During testing, the ground-truth caption is unavailable. However, because the MTM has been trained to map images into the textual subspace, it can provide the language decoder with a "textual-like" starting point based solely on the visual input. This agnostic design allows the MTM to be integrated as a "plug-and-play" module into various architectures, from simple baselines like ShowTell to more complex attention-based models like Bottom-Up Top-Down (UpDown).

Experimental Results

The researchers evaluated the MTM framework on the MS-COCO Captioning dataset using the standard "Karpathy" split. The results demonstrate that the MTM consistently improves performance across multiple metrics and architectures.

Quantitative comparison of MTM-enhanced models against various state-of-the-art baselines on the MS-COCO dataset.

As shown in the results table, the best-performing model, UpDown_MT_RL (incorporating the MTM and reinforcement learning fine-tuning), achieved a CIDEr score of 124.14. This represents a 3.4% improvement over the previous state-of-the-art UpDown_RL (120.10). Even when applied to the simpler ShowTell baseline, the MTM increased the CIDEr score from 96.29 to 100.42, highlighting its robustness.

Qualitatively, MTM-enhanced models show a superior ability to identify fine details and specific human-object interactions. For example, where a baseline might generate a generic description, the MTM model correctly identifies "bottles of wine" or specifies the presence of "two computer monitors."

Conclusion

By explicitly addressing the modality gap through the Modality Transition Module, this research provides a more effective way to initialize language decoders with visually-derived semantic information. The MTM’s ability to capture global semantics rather than isolated objects leads to more descriptive and accurate image captions. Future work may explore applying this modality transition approach to other cross-modal tasks, such as Video Captioning or Visual Question Answering (VQA).

Summary updated

Abstract

Image captioning model is a cross-modality knowledge discovery task, which targets at automatically describing an image with an informative and coherent sentence. To generate the captions, the previous encoder-decoder frameworks directly forward the visual vectors to the recurrent language model, forcing the recurrent units to generate a sentence based on the visual features. Although these sentences are generally readable, they still suffer from the lack of details and highlights, due to the fact that the substantial gap between the image and text modalities is not sufficiently addressed. In this work, we explicitly build a Modality Transition Module (MTM) to transfer visual features into semantic representations before forwarding them to the language model. During the training phase, the modality transition network is optimised by the proposed modality loss, which compares the generated preliminary textual encodings with the target sentence vectors from a pre-trained text auto-encoder. In this way, the visual vectors are transited into the textual subspace for more contextual and precise language generation. The novel MTM can be incorporated into most of the existing methods. Extensive experiments have been conducted on the MS-COCO dataset demonstrating the effectiveness of the proposed framework, improving the performance by 3.4% comparing to the state-of-the-arts.
BibTeX
@misc{wang2021enhanced,
  author = {Ziwei Wang and Yadan Luo and Zi Huang},
  title = {{Enhanced Modality Transition for Image Captioning}},
  year = {2021},
  eprint = {2102.11526},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.11526},
}
Figure