Question-controlled Text-aware Image Captioning

Anwen Hu, Shizhe Chen, Qin Jin

Introduction

Standard image captioning models typically focus on describing visual objects within a scene. While newer text-aware models have expanded this capability by incorporating scene text (e.g., street signs or book titles), they often struggle with information overload. In text-rich environments, these models frequently produce static captions that dump all detected text into a single sentence. For visually impaired users, this approach is often tedious and unhelpful.

To address this, we introduce Question-controlled Text-aware Image Captioning (Qc-TextCap). This task facilitates progressive disclosure, allowing users to first receive a general summary and then use natural language questions to guide the model toward specific details. Unlike Text-VQA, which provides short, single-word answers, Qc-TextCap generates fluent, descriptive sentences tailored to the user's inquiry.

Comparison of TextCap, TextVQA, and the proposed Qc-TextCap task.

The Geometry and Question Aware Model (GQAM)

To solve the Qc-TextCap task, we propose the Geometry and Question Aware Model (GQAM). The model is designed to integrate visual features, spatial relationships, and linguistic control signals through three primary components:

  1. Geometry-informed Visual Encoder: Standard models often treat object regions and OCR (Optical Character Recognition) tokens as independent. GQAM utilizes a novel geometry self-attention mechanism to fuse these features based on their spatial proximity, ensuring the model understands, for example, that a specific number is physically printed on a specific jersey.
  2. Question-guided Encoder: This module employs cross-attention to dynamically select visual features relevant to the user's question. It ensures the model focuses on the correct image regions (e.g., looking at a "jersey" object when the question mentions "jersey").
  3. Multimodal Decoder: A transformer-based decoder equipped with a pointer network. it sequentially generates the final caption by synthesizing geometry-aware visual features, question embeddings, and an initial coarse caption.
The GQAM architecture featuring geometry-informed and question-guided encoders.

Dataset Construction

Since no datasets existed for this interactive task, we developed an automated pipeline to construct two new benchmarks from existing text-aware data: ControlTextCaps (web images) and ControlVizWiz (images taken by blind users).

Our construction method involved using NLP dependency parsing to remove text-specific details from ground-truth captions to create "initial" coarse captions. We then utilized a T5 model trained on SQuAD to reverse-engineer question-answer pairs based on the removed text, creating a robust training set for progressive disclosure.

Experimental Results

GQAM demonstrates superior performance across both quantitative and qualitative metrics. On the ControlTextCaps dataset, GQAM achieved a CIDEr score of 231.74, significantly outperforming the controllable baseline (ControlM4CC) at 215.45. Similar gains were observed on ControlVizWiz (201.35 vs. 195.94).

Ablation studies confirmed that the Geometry-informed encoder is a critical driver of performance; its removal resulted in a drop of up to 14 CIDEr points. Furthermore, human evaluations showed a strong preference for GQAM, with evaluators favoring its accuracy in providing scene text information 43.48% of the time, compared to only 14.23% for the baseline.

Conclusion and Future Work

GQAM represents a significant step toward personalized, interactive assistance for visually impaired users. By using natural language as a control signal, the model provides a more intuitive interface than previous methods relying on bounding boxes. Future work will focus on improving robustness to noisy object detection in low-quality images and integrating external background knowledge to better interpret the context of scene text.

Summary updated

Abstract

For an image with multiple scene texts, different people may be interested in different text information. Current text-aware image captioning models are not able to generate distinctive captions according to various information needs. To explore how to generate personalized text-aware captions, we define a new challenging task, namely Question-controlled Text-aware Image Captioning (Qc-TextCap). With questions as control signals, this task requires models to understand questions, find related scene texts and describe them together with objects fluently in human language. Based on two existing text-aware captioning datasets, we automatically construct two datasets, ControlTextCaps and ControlVizWiz to support the task. We propose a novel Geometry and Question Aware Model (GQAM). GQAM first applies a Geometry-informed Visual Encoder to fuse region-level object features and region-level scene text features with considering spatial relationships. Then, we design a Question-guided Encoder to select the most relevant visual features for each question. Finally, GQAM generates a personalized text-aware caption with a Multimodal Decoder. Our model achieves better captioning performance and question answering ability than carefully designed baselines on both two datasets. With questions as control signals, our model generates more informative and diverse captions than the state-of-the-art text-aware captioning model. Our code and datasets are publicly available at https://github.com/HAWLYQ/Qc-TextCap.
BibTeX
@inproceedings{hu2021question,
  author = {Anwen Hu and Shizhe Chen and Qin Jin},
  title = {{Question-controlled Text-aware Image Captioning}},
  booktitle = {ACM Multimedia},
  year = {2021},
  doi = {10.1145/3474085.3475452},
}
Figure