TL;DR
This paper introduces UNISON, a novel framework for generating image captions in a target language (e.g., Chinese) without requiring any paired image-caption data in that language. It achieves this by leveraging a parallel corpus of sentences and a scene graph representation to bridge the cross-lingual gap and align visual and textual modalities.
What Problem Does This Paper Address?
Traditional image captioning models rely heavily on large datasets of images paired with their corresponding captions. This approach is effective for languages like English, where such datasets are abundant. However, building these datasets for the thousands of other languages spoken worldwide is prohibitively expensive and time-consuming, limiting the accessibility of image captioning technology to a global audience. Existing "unpaired" methods often still rely on some form of caption corpus or pivot languages, which can be restrictive. This paper tackles the challenge of truly unpaired cross-lingual image captioning, aiming to generate captions in a target language without any direct image-caption supervision in that language.
Methods
UNISON employs a two-phase approach:
- Cross-lingual Auto-Encoding: This phase focuses on translating the meaning of a sentence from a source language (English) to a target language (Chinese) using a scene graph representation.
- Scene Graph Extraction: Both source language sentences and images are converted into scene graphs, which are structured representations of objects, their attributes, and their relationships.
- Cross-lingual Hierarchical Graph Mapping (HGM): A novel HGM module is proposed to map scene graphs from the source language's embedding space to the target language's embedding space. This mapping operates at multiple levels: word-level, sub-graph level, and full-graph level, fusing information adaptively. This is crucial for capturing contextual semantics that simple word-to-word translation might miss.
- Auto-Encoding: The mapped scene graph is then encoded and decoded to generate a sentence in the target language. This phase is trained using a parallel corpus of sentences (e.g., English-Chinese translations).
- Unsupervised Cross-modal Feature Mapping: This phase bridges the gap between the visual modality (images) and the textual modality (language).
- Image Scene Graph to Language Scene Graph: Image scene graphs are mapped to the target language's scene graph embedding space using the HGM module (shared from phase 1).
- CycleGAN-based Alignment: A CycleGAN-like approach is used to learn mappings between encoded image scene graph features and encoded sentence scene graph features in an unsupervised manner. This allows the model to align visual information with textual representations without paired image-caption data.
The overall framework is trained using a combination of losses, including a cross-lingual auto-encoding loss (LXE) and a KL divergence loss to encourage alignment of latent scene graph embeddings, and a CycleGAN loss for cross-modal alignment.
Datasets
- Parallel Corpus (for Phase 1): Existing machine translation datasets like WMT19, AIC_MT, UM, and Trans-zh are used to create a paired English-Chinese sentence corpus.
- Image Dataset (for Phase 2): MS-COCO images are used for the unsupervised cross-modal feature mapping phase.
- Evaluation: The COCO-CN dataset (Chinese captions for MS-COCO images) is used for evaluating the final caption generation performance.
Key Insights and Ideas
- Scene Graphs as a Universal Representation: The paper highlights the power of scene graphs as an intermediate representation that can capture semantic meaning independent of language, facilitating cross-lingual transfer.
- Hierarchical Cross-lingual Mapping: The HGM's multi-level approach (word, sub-graph, full-graph) is key to effectively transferring semantic information across languages, going beyond simple word embeddings.
- Unsupervised Alignment is Sufficient: The success of the CycleGAN-based cross-modal mapping demonstrates that explicit image-caption pairs are not strictly necessary for aligning visual and textual features, as long as a rich semantic representation like scene graphs is used.
- Translation at the Graph Level is Superior: The paper argues that translating at the scene graph level is more robust than translating at the sentence level, as it preserves more structural and relational information.
Key Results
- Superior Performance without Caption Corpus: UNISON significantly outperforms the unpaired baseline "Graph-Align" (which relies on translation of generated English captions) on Chinese caption generation, achieving better BLEU, METEOR, ROUGE, and CIDEr scores.
- Competitive with Supervised Methods: Even though UNISON is an unpaired method, it achieves competitive results compared to supervised methods that use translated captions or pseudo-caption pairs. For example, UNISON achieves 63.4 B@1, 43.2 B@2, 29.5 B@3, 17.9 B@4, 45.1 METEOR, and 53.5 CIDEr on the COCO-CN test split, outperforming some supervised baselines.
- Effectiveness of HGM Components: Ablation studies show that incorporating sub-graph and full-graph mapping, as well as the self-gated fusion mechanism, significantly boosts performance.
- Data Augmentation Benefits: Augmenting the sentence scene graphs leads to richer representations and improved captioning performance.
- Human Evaluation: Human evaluations confirm that UNISON generates relevant and fluent captions, scoring higher than baseline models.
Limitations and Future Work
- Reliance on Parallel Sentence Corpus: While UNISON doesn't require paired image-caption data, it still relies on a parallel sentence corpus for the cross-lingual auto-encoding phase. The authors suggest that future work could explore methods that further relax this requirement.
- Scene Graph Generation Quality: The performance of the system is inherently tied to the quality of the scene graph generation. Noisy scene graphs, especially with repeated relation triples, can degrade performance, although the self-gated fusion mechanism helps mitigate this.
- Potential for Further Improvement: The authors express hope that their work will inspire further research in unpaired cross-lingual image captioning. Future directions could involve exploring more sophisticated cross-lingual alignment techniques or end-to-end training paradigms.
Summary updated