Abstract
Training large-scale image captioning (IC) models demands access to a rich and diverse set of training examples, gathered from the wild, often from noisy alt-text data. However, recent modeling approaches to IC often fall short in terms of performance in this case, because they assume a clean annotated dataset and employ an end-to-end generation approach, which often lacks both controllability and interpretability. We address these problems by breaking down the task into two simpler, more controllable tasks: skeleton prediction and skeleton-based caption generation. Specifically, we show that selecting content words as skeletons helps in generating improved and denoised captions when leveraging rich yet noisy alt-text-based uncurated datasets. We also show that the predicted English skeletons can be further cross-lingually leveraged to generate non-English captions, covering French, Italian, German, Spanish, and Hindi. Furthermore, skeleton-based prediction allows for better control of certain caption properties, such as length, content, and gender expression, providing a handle to perform human-in-the-loop semi-automatic corrections.
Introduction: The Challenge of Noisy Alt-Text
Standard image captioning models typically rely on massive datasets of images paired with human-written descriptions. To scale these models, researchers often harvest "alt-text" from the web. While abundant, alt-text is notoriously noisy; instead of describing the visual content of a photo, it often contains metadata or web-specific intent like "house for sale" or "click here for more."
End-to-end models trained on this data often "hallucinate" or repeat irrelevant web context because they cannot distinguish between a description of the image and the intent of the webpage. Moreover, these "black box" models offer little control over output style or inherent biases without expensive retraining.
The Skeleton Framework: A Two-Stage Approach
To address these issues, the researchers propose a two-stage framework that introduces an intermediate, interpretable layer: The Skeleton.
- Stage 1: Skeleton Prediction (Img2Ske): Instead of generating a full sentence immediately, the model predicts a sequence of lemmatized nouns and verbs (e.g., "collection," "book"). By restricting the output to key content words, the model naturally filters out the "meta-noise" common in alt-text.
- Stage 2: Skeleton-based Generation: A second model takes the image and the predicted skeleton to generate a fluent, descriptive caption. This stage ensures the final output is grammatically correct while remaining anchored to the visual objects identified in the skeleton.
Model Architecture
The authors implemented this framework using a Transformer-based architecture with three primary variants for integrating the skeleton information:
- SkeEncoding: The skeleton is processed by a textual encoder and fed into the visual encoder as side information.
- SkeDecoding: The skeleton is prepended to the caption within the decoder.
- SkeAE (Autoencoding): A hybrid approach that simultaneously encodes and decodes the skeleton to reinforce content selection.
Key Insights and Controllability
English as an Interlingua
A significant finding of the study is the utility of English skeletons as a cross-lingual bridge. By predicting an English skeleton and using it to generate captions in other languages (like Hindi or Italian), the model leverages the massive volume of English training data to improve performance in lower-resource languages.
Human-in-the-Loop Correction
Because the skeleton is a simple list of words, it provides a "handle" for intervention. The researchers demonstrated that by automatically replacing gendered terms in the skeleton (e.g., changing "man" to "person"), they could reduce gendered output by 46% in English and 70% in Hindi without any retraining of the underlying models.
Results and Evaluation
In side-by-side human evaluations on the T2 test set, the SkeEncoding model outperformed the standard baseline by a substantial margin of +11.0 gain. The approach significantly shifted caption distributions toward "Visible" relations (describing what is actually in the image) and away from "Meta" or "Story" relations (contextual fluff).
While the model showed gains in 4 out of 5 tested languages, a slight performance dip in Spanish highlighted the sensitivity of these models to regional dialect mismatches between training data and human evaluators.
Conclusion
By breaking image captioning into content selection and surface realization, this research provides a robust method for denoising web-scale data. The skeleton-based approach not only improves caption quality but also introduces much-needed interpretability and a mechanism for mitigating model bias in a multi-lingual context.
Summary updated