Evaluating and Complementing Vision-to-Language Technology for People who are Blind with Conversational Crowdsourcing

Elliot Salisbury, Ece Kamar, Meredith Ringel Morris

Introduction

Social media platforms are increasingly visual, yet accessibility for blind and visually impaired (BVI) users remains a significant challenge. While automated "alt text" systems have been introduced, these AI-generated captions often lack the specific context required to understand the intent of a post. For instance, an AI might identify "a person holding a sign" but fail to capture the text on the sign or the reason the image is relevant to the accompanying tweet.

An example of a tweet where automated captioning fails to capture the necessary context.

The researchers identify three primary hurdles: AI inaccuracy, a persistent context gap where APIs ignore accompanying text, and the lack of scalability for high-quality human assistance.

Methodology

The study compared four distinct workflows for generating image descriptions:

  1. Vision-to-Language (Baseline): Fully automated captions via Microsoft’s CaptionBot API.
  2. Human-Corrected Captions: Sighted crowd workers editing AI captions while viewing the tweet text.
  3. TweetTalk (Conversational): A real-time chat interface where a BVI user asks a sighted assistant questions about the image.
  4. Structured Q&A: A non-real-time workflow where sighted workers answer predetermined questions based on common BVI inquiries (e.g., "What are the subjects doing?", "Is there humor?").

The researchers utilized a dataset of 85 tweets from high-profile accounts and trending hashtags to represent the complex, "in-the-wild" nature of social media imagery, which includes memes and news photos that are notoriously difficult for AI to describe.

Key Insights

A significant finding was the "anchor effect" of inaccurate AI. If an initial AI caption is incorrect, BVI users often struggle to reach a correct understanding even after subsequent human intervention. Furthermore, the study highlighted that "context is king." Unlike AI, humans use the tweet's text to prioritize information—such as describing political slogans rather than just "a group of people." By analyzing these interactions, the authors identified that BVI users consistently seek specific information (who, what, where, and the "vibe"), leading to the development of the more efficient Structured Q&A model.

Results and Analysis

The evaluation showed that human-in-the-loop workflows significantly outperformed pure AI. The Structured Q&A workflow achieved the highest satisfaction factor (0.68), while the baseline AI captions resulted in a negative satisfaction score (-1.06), indicating that users felt misled once the true content was revealed.

Quantitative comparison of user satisfaction across different captioning workflows.

While the TweetTalk conversational model provided high-quality results, it was more expensive ($0.95/tweet) and slower (8 minutes) than the Structured Q&A ($1.20/tweet), which offers better robustness through asynchronous processing.

Limitations and Future Directions

The study primarily relied on sighted users simulating BVI perspectives, though findings were validated with a small group of BVI participants. Scaling human-in-the-loop descriptions to the entire internet remains cost-prohibitive; however, the authors suggest focusing on viral content and using human-generated Q&A data to train more nuanced future AI systems.

Summary updated

Abstract

We study how real-time crowdsourcing can be used both for evaluating the value provided by existing automated approaches and for enabling workflows that provide scalable and useful alt text to blind users. We show that the shortcomings of existing AI image captioning systems frequently hinder a user's understanding of an image they cannot see to a degree that even clarifying conversations with sighted assistants cannot correct. Based on analysis of clarifying conversations collected from our studies, we design experiences that can effectively assist users in a scalable way without the need for real-time interaction. Our results provide lessons and guidelines that the designers of future AI captioning systems can use to improve labeling of social media imagery for blind users.
BibTeX
@inproceedings{salisbury2018evaluating,
  author = {Elliot Salisbury and Ece Kamar and Meredith Ringel Morris},
  title = {{Evaluating and Complementing Vision-to-Language Technology for People who are Blind with Conversational Crowdsourcing}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/751},
}
Figure