Introduction
Standard image captioning models typically focus on describing visual objects within a scene. While newer text-aware models have expanded this capability by incorporating scene text (e.g., street signs or book titles), they often struggle with information overload. In text-rich environments, these models frequently produce static captions that dump all detected text into a single sentence. For visually impaired users, this approach is often tedious and unhelpful.
To address this, we introduce Question-controlled Text-aware Image Captioning (Qc-TextCap). This task facilitates progressive disclosure, allowing users to first receive a general summary and then use natural language questions to guide the model toward specific details. Unlike Text-VQA, which provides short, single-word answers, Qc-TextCap generates fluent, descriptive sentences tailored to the user's inquiry.
The Geometry and Question Aware Model (GQAM)
To solve the Qc-TextCap task, we propose the Geometry and Question Aware Model (GQAM). The model is designed to integrate visual features, spatial relationships, and linguistic control signals through three primary components:
- Geometry-informed Visual Encoder: Standard models often treat object regions and OCR (Optical Character Recognition) tokens as independent. GQAM utilizes a novel geometry self-attention mechanism to fuse these features based on their spatial proximity, ensuring the model understands, for example, that a specific number is physically printed on a specific jersey.
- Question-guided Encoder: This module employs cross-attention to dynamically select visual features relevant to the user's question. It ensures the model focuses on the correct image regions (e.g., looking at a "jersey" object when the question mentions "jersey").
- Multimodal Decoder: A transformer-based decoder equipped with a pointer network. it sequentially generates the final caption by synthesizing geometry-aware visual features, question embeddings, and an initial coarse caption.
Dataset Construction
Since no datasets existed for this interactive task, we developed an automated pipeline to construct two new benchmarks from existing text-aware data: ControlTextCaps (web images) and ControlVizWiz (images taken by blind users).
Our construction method involved using NLP dependency parsing to remove text-specific details from ground-truth captions to create "initial" coarse captions. We then utilized a T5 model trained on SQuAD to reverse-engineer question-answer pairs based on the removed text, creating a robust training set for progressive disclosure.
Experimental Results
GQAM demonstrates superior performance across both quantitative and qualitative metrics. On the ControlTextCaps dataset, GQAM achieved a CIDEr score of 231.74, significantly outperforming the controllable baseline (ControlM4CC) at 215.45. Similar gains were observed on ControlVizWiz (201.35 vs. 195.94).
Ablation studies confirmed that the Geometry-informed encoder is a critical driver of performance; its removal resulted in a drop of up to 14 CIDEr points. Furthermore, human evaluations showed a strong preference for GQAM, with evaluators favoring its accuracy in providing scene text information 43.48% of the time, compared to only 14.23% for the baseline.
Conclusion and Future Work
GQAM represents a significant step toward personalized, interactive assistance for visually impaired users. By using natural language as a control signal, the model provides a more intuitive interface than previous methods relying on bounding boxes. Future work will focus on improving robustness to noisy object detection in low-quality images and integrating external background knowledge to better interpret the context of scene text.
Summary updated