Text Conditional Alt-Text Generation for Twitter Images

Nikita Srivatsan, Sofía Samaniego, Omar Florez, Taylor Berg-Kirkpatrick

Introduction

Researchers from CMU, UCSD, and LatinX in AI have developed a multimodal model that generates high-quality alt-text for Twitter images by leveraging the surrounding tweet as context. For the blind and low-vision (BLV) community, social media is often a "visual black hole." Roughly 98% of images are uploaded without any description, leaving screen readers to simply announce "Image," providing zero context.

Standard image captioning models often fail because they provide literal descriptions in a vacuum. Social media images are frequently tied to the post's text. As shown in Figure 4, a stack of papers might be identified as "books" by a standard model, whereas the tweet context identifies them as "election flyers."

Comparison of standard image captioning versus text-conditioned alt-text generation and dataset differences.

Methodology

The proposed architecture builds upon ClipCap with a significant modification to include textual context.

  1. Visual Path: A pretrained CLIP encoder extracts visual features, which are then passed through a multi-layer perceptron "mapping network" to translate them into the word embedding space.
  2. Contextual Path: The model simultaneously processes the tweet text into embeddings.
  3. Multimodal Prefix: The core innovation is the concatenation of these signals. The model creates a "prefix" for a large language model (GPT-2) consisting of the projected image features and the tweet text.
  4. Generation: GPT-2 uses this combined prefix as a prompt to autoregressively generate the alt-text.
Architecture diagram showing the integration of image encoding and tweet text into a multimodal prefix for GPT-2.

Dataset

The researchers introduced a massive dataset of 371k Twitter images paired with original tweet text and user-written alt-text. Unlike existing datasets like "Conceptual Captions," which are often short and colloquial, this dataset features longer, more functional descriptions (averaging 40 tokens). To protect user privacy, the data was de-identified and released in a "dehydrated" format.

Key Insights and Results

The study found that visual and textual signals are complementary, a phenomenon termed "information stacking." This allows the model to resolve ambiguities that neither source could handle alone. Furthermore, the model respects the functional nature of alt-text over the subjective nature of typical captions.

Quantitatively, the "Text + Image" model significantly outperformed baselines. On the BLEU@4 metric, it scored 1.826, compared to 0.111 for a frozen BLIP-2 and 0.681 for a finetuned ClipCap without text conditioning. Human evaluations confirmed these findings, with annotators preferring the multimodal model's output in two-thirds of cases due to better fluency and descriptiveness.

Limitations and Future Work

Challenges remain, including noisy training labels from users and a lack of dedicated OCR for transcribing text within images. Some users also misuse the alt-text field for jokes rather than descriptions. Future work aims to use this model as an "assistant" to help sighted users draft alt-text, improving overall social media accessibility.

Summary updated

BibTeX
@inproceedings{srivatsan2024text,
  author = {Nikita Srivatsan and Sofía Samaniego and Omar Florez and Taylor Berg-Kirkpatrick},
  title = {{Text Conditional Alt-Text Generation for Twitter Images}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=97Dl82avFs},
}
Figure