COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrieval

Xirong Li, Chaoxi Xu, Xiaoxu Wang, Weiyu Lan, Zhengxiong Jia, Gang Yang, Jieping XuarXiv 2018 · arXiv

Here is a concise, engaging summary of the paper designed for researchers and engineers interested in computer vision and natural language processing.

TL;DR

This paper introduces COCO-CN, a new dataset that extends the popular MS-COCO benchmark by adding manually written Chinese captions and tags for over 20,000 images. To scale up data collection without sacrificing quality, the authors designed an AI-driven, recommendation-assisted annotation system. Along with the dataset, the paper establishes strong, conceptually simple baseline models that set a new standard for cross-lingual image tagging, captioning, and retrieval.

What Problem Does This Paper Address?

While deep learning has driven massive progress in vision-language tasks like image captioning and retrieval, the vast majority of large-scale datasets (such as ImageNet and MS-COCO) are exclusively in English. Non-English datasets are rare, and the few that exist suffer from major drawbacks: they are either small, generated via flawed machine translations, or heavily biased toward specific domains (e.g., only describing human activities).

This lack of high-quality, diverse bilingual data creates a bottleneck for developing models that can serve the majority of the world's population who do not speak English. The authors address this gap by providing a high-quality, manually curated Chinese-English dataset alongside unified baselines to evaluate cross-lingual performance.

Methods

The authors developed a dual-pronged approach encompassing an intelligent data collection pipeline and specialized cross-lingual model architectures:

  1. Recommendation-Assisted Collective Annotation: To build the dataset efficiently, the authors built a web-based annotation system. Instead of writing descriptions from scratch, human annotators are presented with the top 5 tags and top 5 sentences automatically recommended by AI models based on the image content. Annotators can instantly accept or edit these suggestions.
  2. Cascading MLP for Image Tagging: To predict Chinese tags, the model first maps visual features to a 512-dimensional English semantic space (predicting English tags). It then concatenates these English semantic features with the original visual features to predict the final Chinese tags.
  3. Sequential Learning for Image Captioning: Rather than simply mixing machine-translated text with high-quality manual text, the authors use a two-stage training strategy. The captioning model is first pre-trained on a large corpus of machine-translated sentences (COCO-MT) to learn a broad vocabulary, and then fine-tuned on the smaller, high-quality manual COCO-CN dataset.
  4. Enhanced Word2VisualVec (W2VV) for Retrieval: For cross-lingual image retrieval, the authors upgraded the standard W2VV architecture by adding a soft attention mechanism over the Gated Recurrent Unit (GRU) text encoder and replacing the standard Mean Squared Error (MSE) loss with a Contrastive Loss function to better handle negative image-text pairs.
Fig. 1. The development of COCO-CN and its usage in three tasks.

Datasets

The experiments utilize several key datasets:

  • COCO-CN (New): 20,342 images from MS-COCO enriched with 27,218 manual Chinese sentences and 70,993 tags. It is split into 18,342 training, 1,000 validation, and 1,000 test images.
  • COCO-MT: A machine-translated version of the full MS-COCO dataset (via Baidu Translation), containing over 121,000 training images.
  • Flickr8k-CN & AIC-ICC: Existing Chinese image-text datasets used as baseline comparisons.
  • NUS-WIDE100: An independent test set of 100 images used to verify model generalization and prevent dataset bias during human evaluation.

Key Insights and Ideas

  • Humans Don't Just Translate: A comparison revealed that manual Chinese captions introduce 45.3% novel keywords compared to literal machine translations of the original English captions. Humans describe images from unique, subjective, and personalized viewpoints rather than performing direct word-to-word translation.
  • AI Assistance Speeds Up Annotation: The recommendation system achieved a 54% acceptance rate for suggested tags. Furthermore, measuring the Levenshtein distance (edit distance) proved that sentence recommendations significantly reduced the manual typing effort required by human annotators.
Fig. 2. A snapshot of our web based image annotation system. Rather than starting from scratch, an annotator is provided with five tags and five sentences automatically recommended by the system based on the pictorial content. When clicked, the tags and sentences will instantly appear in the editable text forms to assist manual annotation. Source code of the annotation system is publicly available at https://tinyurl.com/cococn-system.
  • Data Quality Beats Data Quantity: Models trained on the smaller, high-quality COCO-CN dataset consistently outperformed models trained on the much larger, machine-translated COCO-MT dataset, highlighting the value of clean, human-curated data.

Key Results

  • Image Tagging: The proposed Cascading MLP achieved the highest F-measure (0.517 in automated testing, 0.623 in human evaluation), outperforming commercial alternatives like the Clarifai API.
  • Image Captioning: The Sequential Learning strategy achieved a remarkable CIDEr score of 98.4, vastly superior to models trained on Flickr8k-CN (22.9) or the heavily biased AIC-ICC dataset (24.6).
  • Image Retrieval: The Enhanced W2VV model achieved a Mean Reciprocal Rank (MRR) of 0.511 when combining cross-lingual and cross-modal matching, representing a 6.2% relative improvement over the original baseline architecture.

Limitations and Future Work

The primary limitation acknowledged by the authors is the dataset's scale. At roughly 20,000 images, COCO-CN is smaller than massive monolingual datasets because the authors prioritized strict quality control over cheap, unvetted crowd-sourcing.

Moving forward, the authors suggest scaling up the dataset using their annotation pipeline and exploring more advanced algorithmic architectures—such as adapting state-of-the-art monolingual transformers and attention models—specifically tailored for the cross-lingual setting.

The dataset and code are publicly available at https://github.com/li-xirong/coco-cn.

Summary updated

Abstract

This paper contributes to cross-lingual image annotation and retrieval in terms of data and baseline methods. We propose COCO-CN, a novel dataset enriching MS-COCO with manually written Chinese sentences and tags. For more effective annotation acquisition, we develop a recommendation-assisted collective annotation system, automatically providing an annotator with several tags and sentences deemed to be relevant with respect to the pictorial content. Having 20,342 images annotated with 27,218 Chinese sentences and 70,993 tags, COCO-CN is currently the largest Chinese-English dataset that provides a unified and challenging platform for cross-lingual image tagging, captioning and retrieval. We develop conceptually simple yet effective methods per task for learning from cross-lingual resources. Extensive experiments on the three tasks justify the viability of the proposed dataset and methods. Data and code are publicly available at https://github.com/li-xirong/coco-cn
BibTeX
@misc{li2018coco,
  author = {Xirong Li and Chaoxi Xu and Xiaoxu Wang and Weiyu Lan and Zhengxiong Jia and Gang Yang and Jieping Xu},
  title = {{COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrieval}},
  year = {2018},
  eprint = {1805.08661},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.08661},
}

Loading related content...

Figure