Here is a concise, engaging summary of the paper designed for researchers and engineers interested in computer vision and natural language processing.
TL;DR
This paper introduces COCO-CN, a new dataset that extends the popular MS-COCO benchmark by adding manually written Chinese captions and tags for over 20,000 images. To scale up data collection without sacrificing quality, the authors designed an AI-driven, recommendation-assisted annotation system. Along with the dataset, the paper establishes strong, conceptually simple baseline models that set a new standard for cross-lingual image tagging, captioning, and retrieval.
What Problem Does This Paper Address?
While deep learning has driven massive progress in vision-language tasks like image captioning and retrieval, the vast majority of large-scale datasets (such as ImageNet and MS-COCO) are exclusively in English. Non-English datasets are rare, and the few that exist suffer from major drawbacks: they are either small, generated via flawed machine translations, or heavily biased toward specific domains (e.g., only describing human activities).
This lack of high-quality, diverse bilingual data creates a bottleneck for developing models that can serve the majority of the world's population who do not speak English. The authors address this gap by providing a high-quality, manually curated Chinese-English dataset alongside unified baselines to evaluate cross-lingual performance.
Methods
The authors developed a dual-pronged approach encompassing an intelligent data collection pipeline and specialized cross-lingual model architectures:
- Recommendation-Assisted Collective Annotation: To build the dataset efficiently, the authors built a web-based annotation system. Instead of writing descriptions from scratch, human annotators are presented with the top 5 tags and top 5 sentences automatically recommended by AI models based on the image content. Annotators can instantly accept or edit these suggestions.
- Cascading MLP for Image Tagging: To predict Chinese tags, the model first maps visual features to a 512-dimensional English semantic space (predicting English tags). It then concatenates these English semantic features with the original visual features to predict the final Chinese tags.
- Sequential Learning for Image Captioning: Rather than simply mixing machine-translated text with high-quality manual text, the authors use a two-stage training strategy. The captioning model is first pre-trained on a large corpus of machine-translated sentences (COCO-MT) to learn a broad vocabulary, and then fine-tuned on the smaller, high-quality manual COCO-CN dataset.
- Enhanced Word2VisualVec (W2VV) for Retrieval: For cross-lingual image retrieval, the authors upgraded the standard W2VV architecture by adding a soft attention mechanism over the Gated Recurrent Unit (GRU) text encoder and replacing the standard Mean Squared Error (MSE) loss with a Contrastive Loss function to better handle negative image-text pairs.
Datasets
The experiments utilize several key datasets:
- COCO-CN (New): 20,342 images from MS-COCO enriched with 27,218 manual Chinese sentences and 70,993 tags. It is split into 18,342 training, 1,000 validation, and 1,000 test images.
- COCO-MT: A machine-translated version of the full MS-COCO dataset (via Baidu Translation), containing over 121,000 training images.
- Flickr8k-CN & AIC-ICC: Existing Chinese image-text datasets used as baseline comparisons.
- NUS-WIDE100: An independent test set of 100 images used to verify model generalization and prevent dataset bias during human evaluation.
Key Insights and Ideas
- Humans Don't Just Translate: A comparison revealed that manual Chinese captions introduce 45.3% novel keywords compared to literal machine translations of the original English captions. Humans describe images from unique, subjective, and personalized viewpoints rather than performing direct word-to-word translation.
- AI Assistance Speeds Up Annotation: The recommendation system achieved a 54% acceptance rate for suggested tags. Furthermore, measuring the Levenshtein distance (edit distance) proved that sentence recommendations significantly reduced the manual typing effort required by human annotators.
- Data Quality Beats Data Quantity: Models trained on the smaller, high-quality COCO-CN dataset consistently outperformed models trained on the much larger, machine-translated COCO-MT dataset, highlighting the value of clean, human-curated data.
Key Results
- Image Tagging: The proposed Cascading MLP achieved the highest F-measure (0.517 in automated testing, 0.623 in human evaluation), outperforming commercial alternatives like the Clarifai API.
- Image Captioning: The Sequential Learning strategy achieved a remarkable CIDEr score of 98.4, vastly superior to models trained on Flickr8k-CN (22.9) or the heavily biased AIC-ICC dataset (24.6).
- Image Retrieval: The Enhanced W2VV model achieved a Mean Reciprocal Rank (MRR) of 0.511 when combining cross-lingual and cross-modal matching, representing a 6.2% relative improvement over the original baseline architecture.
Limitations and Future Work
The primary limitation acknowledged by the authors is the dataset's scale. At roughly 20,000 images, COCO-CN is smaller than massive monolingual datasets because the authors prioritized strict quality control over cheap, unvetted crowd-sourcing.
Moving forward, the authors suggest scaling up the dataset using their annotation pipeline and exploring more advanced algorithmic architectures—such as adapting state-of-the-art monolingual transformers and attention models—specifically tailored for the cross-lingual setting.
The dataset and code are publicly available at https://github.com/li-xirong/coco-cn.
Summary updated