Introduction
Traditional image captioning systems rely heavily on massive datasets of images paired with human-written descriptions, such as MS-COCO. However, scaling this approach to the world’s 7,000+ languages is prohibitively expensive. While "unpaired" methods have emerged, they typically still require a large monolingual caption corpus in the target language to learn the data distribution.
UNISON (UNpaIred crosS-lingual image captiONing) addresses a more challenging "zero-caption" scenario: building a captioner for a language where parallel text (bitext) and images are available, but actual image-caption pairs in the target language are entirely absent. The framework leverages Scene Graphs as an intermediary structural bridge to map visual relationships to linguistic structures.
The UNISON Framework
The core philosophy of UNISON is that the "internal logic" of a scene—objects, their attributes, and their relationships—is largely universal across languages. By using scene graphs as a pivot modality, the model can bridge the gap between vision and language more stably than raw pixels or unstructured embeddings.
The framework operates in two distinct phases:
- Cross-lingual Auto-encoding: Using a parallel sentence corpus (e.g., English-Chinese), the model learns a Hierarchical Graph Mapping (HGM). It extracts a scene graph from an English sentence and learns to map it into a latent space that can decode into a Chinese sentence. HGM operates at three levels—word, sub-graph, and full-graph—to ensure that the context of individual nodes is preserved through their structural neighbors.
- Unsupervised Cross-modal Feature Mapping: Once the model can translate from "Graph Space" to the "Target Language," it must learn to map from "Image" to "Graph Space." This is achieved using an unsupervised alignment approach (similar to CycleGAN) that enforces cycle consistency (Image -> Language -> Image) to ensure mapping accuracy.
Key Insights: Hierarchical Mapping and Fusion
Simple word-to-word translation of graph nodes often fails because it ignores syntax and context. UNISON captures "structural semantics" by mapping sub-graphs and full graphs. To combine these levels effectively, the model employs a self-gated fusion mechanism that adaptively decides which level of information (word, sub-graph, or full-graph) is most critical for a given context.
To further improve robustness, the authors used back-translation to enrich English scene graphs, increasing the complexity and density of the training data. A joint-training mechanism using KL-divergence also ensures that encoded features of source and target languages remain semantically close in the latent space.
Experimental Results
UNISON was evaluated on Chinese image captioning using the COCO-CN dataset. The model demonstrated significant improvements over existing baselines in the "no caption corpus" setting.
Key performance highlights include:
- Superiority over Pivot Methods: UNISON achieved a 12.7 CIDEr score, significantly outperforming the state-of-the-art unpaired method (Graph-Align) combined with Google Translate (9.3 CIDEr).
- Competitive with Supervised Pipelines: Despite being an unpaired method, UNISON's results (17.9 BLEU-4) were competitive with supervised pipelines that utilize English-to-Chinese pseudo-captions.
- Human Evaluation: In trials measuring fluency and relevancy, UNISON scored 4.06/5.0 in fluency, approaching the quality of human-written ground truth (4.91/5.0).
Limitations and Future Work
While effective, the system's performance is currently capped by the quality of the initial scene graph parsers for both images and text. Errors in the parser cannot be easily recovered by the captioner. Additionally, image-derived scene graphs are often noisier than text-derived ones, sometimes leading to repeated relations in the output. Future work will focus on refining unsupervised alignment to handle this noise and testing the framework on morphologically rich or lower-resource languages.
Summary updated