Abstract
Visual Question Answering (VQA) often suffers from linguistic ambiguity where a single question can be interpreted in multiple ways, leading to divergent answers. This paper introduces a new dataset and the VT5 generative model to rephrase vague questions into specific ones based on answer clusters. By shifting the focus from treating disagreement as noise to treating it as a communication challenge, the authors demonstrate that models can learn to resolve ambiguity and that internal representations naturally align with human intent.
The Challenge of Linguistic Ambiguity
In standard VQA benchmarks, models are often penalized for providing answers that differ from the ground truth, even when those answers are technically correct under a different interpretation of the question. For instance, the question "What kind of flowers are these?" is inherently vague. One annotator might respond with the species ("daisies"), while another might describe the color ("purple").
Traditionally, this variance is dismissed as annotator noise. However, this research argues that the root cause is often linguistic ambiguity rather than visual underspecification. To address this, the authors propose a system that identifies these distinct "intents" and generates rephrased, unambiguous questions for each.
Methodology: Data and Models
The researchers developed a three-step pipeline to formalize and solve this problem:
- Dataset Curation: Using a subset of VQAv2, the team identified high-disagreement questions. They employed GloVe embeddings and K-means clustering to group similar answers, which were then manually refined by humans to create a dataset of 1,820 image-question-answer triples.
- Ontology of Ambiguity: A taxonomy was built using the PropBank framework, categorizing ambiguities into Property-based (e.g., location, color), Dynamic (e.g., cause, purpose), and Pragmatic (e.g., which specific object in a group is being referenced).
- The VT5 Model: The core of the solution is VT5, a model combining a ViLT (Vision-Language Transformer) encoder with a T5 decoder. VT5 is designed to take an image and a specific answer group as input and generate a clarifying question. To maintain relevance, the model uses constrained decoding to ensure key nouns from the original question are preserved.
Key Insights and Results
A significant finding of this work is the emergence of "intent" within the model's latent space. Even without explicit supervision on question categories, the VT5 model's internal representations naturally clustered answers according to the specific underlying question they addressed.
Quantitatively, the VT5 model achieved an F1 score of 79.0 in clustering answers by intent, significantly outperforming the ViLT baseline (60.1). Human evaluations further confirmed that the rephrased questions were far less ambiguous; judges were much more likely to correctly reject "distractor" answers that were valid for the original vague question but incorrect for the specific rephrased version.
Conclusion and Future Directions
This research shifts the paradigm of VQA from a simple classification task to a more nuanced communication task. By acknowledging and resolving linguistic ambiguity, AI systems can become more interpretable and reliable.
While the current dataset is relatively small (241 unique image-question pairs), it serves as a proof of concept for larger-scale implementations. Future work could see these rephrasing capabilities integrated into interactive digital assistants, allowing them to ask clarifying questions—"Do you mean the color or the species?"—rather than providing a potentially irrelevant answer.
Summary updated