Abstract
Training large-scale image captioning (IC) models demands access to a rich and diverse set of training examples, gathered from the wild, often from noisy alt-text data. However, recent modeling approaches to IC often fall short in terms of performance in this case, because they assume a clean annotated dataset (as opposed to the noisier alt-text--based annotations), and employ an end-to-end generation approach, which often lacks both controllability and interpretability. We address these problems by breaking down the task into two simpler, more controllable tasks -- skeleton prediction and skeleton-based caption generation. Specifically, we show that selecting content words as skeletons helps in generating improved and denoised captions when leveraging rich yet noisy alt-text--based uncurated datasets. We also show that the predicted English skeletons can be further cross-lingually leveraged to generate non-English captions, and present experimental results covering caption generation in French, Italian, German, Spanish and Hindi. We also show that skeleton-based prediction allows for better control of certain caption properties, such as length, content, and gender expression, providing a handle to perform human-in-the-loop semi-automatic corrections.
Introduction
The gold standard for image captioning (IC) involves human-curated datasets like COCO, but these are expensive and difficult to scale. A popular alternative is using "alt-text" scraped from the web (e.g., Conceptual Captions). However, alt-text is notoriously noisy. Instead of describing the image (e.g., "front view of a house"), it often reflects the webpage's intent (e.g., "house for sale"). Standard end-to-end models trained on this data tend to "hallucinate" irrelevant details or produce "meta" captions that don't describe visible objects. This paper aims to bridge this "semantic gap" by separating content selection from sentence realization.
Methods
The authors propose a dual-staged pipeline that breaks the generation process into two distinct steps:
- Skeleton Prediction (Img2Ske): Given an image, the model first predicts a "skeleton"—a sequence of key lemmas (nouns and verbs) derived from the original alt-text using distant supervision (POS tagging).
- Skeleton-based Caption Generation: The predicted skeleton is then used to guide the final caption generation.
The underlying architecture is a Transformer-based encoder-decoder utilizing GraphRISE and Faster R-CNN for sophisticated visual feature extraction. The authors experimented with three architectural variants for the second stage:
- SkeEncoding: The skeleton is fed into the encoder using a "text-as-side" attention mechanism.
- SkeDecoding: The skeleton and caption are concatenated and predicted by the same decoder.
- SkeAE (Autoencoding): A hybrid that simultaneously encodes and decodes the skeleton to provide bidirectional context.
The models were trained on the Conceptual Captions (CC) dataset (3.3M pairs) and evaluated on a multilingual version (French, Italian, German, Spanish, and Hindi) as well as an out-of-domain T2 Test Set for human evaluation.
Key Insights and Results
By forcing the model to first predict a skeleton of nouns and verbs, it effectively filters out web noise (like "click here") and focuses on visible objects. This "bottlenecking" approach significantly shifts the distribution of captions toward visible relationships.
Human Evaluation vs. Automated Metrics
A key finding was the discrepancy between standard metrics and human judgment. While CIDEr scores remained relatively flat across models, human evaluators overwhelmingly preferred the denoised captions produced by the skeleton-based models. In side-by-side evaluations, the SkeEncoding model outperformed the baseline end-to-end model by a relative gain of +11.0 in English.
Multilingual Transfer and Controllability
Interestingly, skeletons predicted in English can successfully guide caption generation in other languages. This "pivoting" allows the model to leverage English linguistic tools to improve performance in low-resource languages like Hindi, which saw a +4.5 gain in human preference. Furthermore, the skeleton acts as a "handle" for controllability. By manually editing the skeleton, users can control caption length, content, and even mitigate gender bias (e.g., changing "man" to "person").
Conclusion
This two-stage approach addresses the noise inherent in web-scale alt-text by separating content selection from realization. While some challenges remain—such as performance dips in specific dialects (e.g., Spanish)—the method provides a robust framework for generating cleaner, more descriptive, and controllable captions. Future work may involve integrating this content selection mechanism into the pre-training phase of even larger multimodal models to automatically "clean" web-scale data.
Summary updated