Unsupervised Cross-lingual Image Captioning

Jiahui Gao, Yi Zhou, Philip L. H. Yu, Jiuxiang Gu

Introduction

Traditional image captioning systems rely heavily on massive datasets of images paired with human-written descriptions, such as MS-COCO. However, scaling this approach to the world’s 7,000+ languages is prohibitively expensive. While "unpaired" methods have emerged, they typically still require a large monolingual caption corpus in the target language to learn the data distribution.

UNISON (UNpaIred crosS-lingual image captiONing) addresses a more challenging "zero-caption" scenario: building a captioner for a language where parallel text (bitext) and images are available, but actual image-caption pairs in the target language are entirely absent. The framework leverages Scene Graphs as an intermediary structural bridge to map visual relationships to linguistic structures.

The UNISON Framework

The core philosophy of UNISON is that the "internal logic" of a scene—objects, their attributes, and their relationships—is largely universal across languages. By using scene graphs as a pivot modality, the model can bridge the gap between vision and language more stably than raw pixels or unstructured embeddings.

The UNISON framework consists of two phases: (a) Cross-Lingual Auto-Encoding and (b) Unsupervised Cross-modal Feature Mapping.

The framework operates in two distinct phases:

  1. Cross-lingual Auto-encoding: Using a parallel sentence corpus (e.g., English-Chinese), the model learns a Hierarchical Graph Mapping (HGM). It extracts a scene graph from an English sentence and learns to map it into a latent space that can decode into a Chinese sentence. HGM operates at three levels—word, sub-graph, and full-graph—to ensure that the context of individual nodes is preserved through their structural neighbors.
  2. Unsupervised Cross-modal Feature Mapping: Once the model can translate from "Graph Space" to the "Target Language," it must learn to map from "Image" to "Graph Space." This is achieved using an unsupervised alignment approach (similar to CycleGAN) that enforces cycle consistency (Image -> Language -> Image) to ensure mapping accuracy.

Key Insights: Hierarchical Mapping and Fusion

Simple word-to-word translation of graph nodes often fails because it ignores syntax and context. UNISON captures "structural semantics" by mapping sub-graphs and full graphs. To combine these levels effectively, the model employs a self-gated fusion mechanism that adaptively decides which level of information (word, sub-graph, or full-graph) is most critical for a given context.

To further improve robustness, the authors used back-translation to enrich English scene graphs, increasing the complexity and density of the training data. A joint-training mechanism using KL-divergence also ensures that encoded features of source and target languages remain semantically close in the latent space.

Experimental Results

UNISON was evaluated on Chinese image captioning using the COCO-CN dataset. The model demonstrated significant improvements over existing baselines in the "no caption corpus" setting.

Qualitative results comparing UNISON (HGM + CMM) against baselines and ground truth captions.

Key performance highlights include:

  • Superiority over Pivot Methods: UNISON achieved a 12.7 CIDEr score, significantly outperforming the state-of-the-art unpaired method (Graph-Align) combined with Google Translate (9.3 CIDEr).
  • Competitive with Supervised Pipelines: Despite being an unpaired method, UNISON's results (17.9 BLEU-4) were competitive with supervised pipelines that utilize English-to-Chinese pseudo-captions.
  • Human Evaluation: In trials measuring fluency and relevancy, UNISON scored 4.06/5.0 in fluency, approaching the quality of human-written ground truth (4.91/5.0).

Limitations and Future Work

While effective, the system's performance is currently capped by the quality of the initial scene graph parsers for both images and text. Errors in the parser cannot be easily recovered by the captioner. Additionally, image-derived scene graphs are often noisier than text-derived ones, sometimes leading to repeated relations in the output. Future work will focus on refining unsupervised alignment to handle this noise and testing the framework on morphologically rich or lower-resource languages.

Summary updated

Abstract

Image captioning has emerged as an interesting research field in recent years due to its broad application scenarios. The traditional paradigm of image captioning relies on paired image-caption datasets to train the model in a supervised manner. However, creating such paired datasets for every target language is prohibitively expensive, which hinders the extensibility of captioning technology and deprives a large part of the world population of its benefit. In this work, we present a novel unpaired cross-lingual method to generate image captions without relying on any caption corpus in the source or the target language. Specifically, our method consists of two phases: (i) a cross-lingual auto-encoding process, which utilizing a sentence parallel (bitext) corpus to learn the mapping from the source to the target language in the scene graph encoding space and decode sentences in the target language, and (ii) a cross-modal unsupervised feature mapping, which seeks to map the encoded scene graph features from image modality to language modality. We verify the effectiveness of our proposed method on the Chinese image caption generation task. The comparisons against several existing methods demonstrate the effectiveness of our approach.
BibTeX
@inproceedings{gao2022unsupervised,
  author = {Jiahui Gao and Yi Zhou and Philip L. H. Yu and Jiuxiang Gu},
  title = {{Unsupervised Cross-lingual Image Captioning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21310},
}
Figure