AmbiSteer: Selectively Eliciting Ambiguity-Aware Responses in Vision-Language Models

Monica Riedler, Samuele Antonelli, Barbara Plank, Raffaella BernardiarXiv 2026 · arXiv

TL;DR

AMBISTEER is a lightweight, inference-time framework that enables Vision-Language Models (VLMs) to handle referential ambiguity selectively. By combining activation steering with a classifier-gated mechanism, it allows models to recognize when a question is unclear and respond by asking for clarification or enumerating possibilities, improving appropriate response rates from 59% to 77% on average.

What Problem Does This Paper Address?

Current Vision-Language Models suffer from "systemic overconfidence." When faced with a referentially ambiguous question—like asking "What is the man holding?" in an image containing three different men—most models will arbitrarily pick one person and provide a firm, single answer.

Ideally, a model should recognize this uncertainty and either ask for clarification ("Which man do you mean?") or provide an exhaustive answer ("The man on the left has a plate, while the man on the right has a bottle"). However, simply prompting a model to always be "ambiguity-aware" leads to a violation of Grice’s maxim of quantity: models start asking for clarification even when the referent is perfectly clear, which is frustrating and inefficient for users. The core challenge is achieving selective ambiguity awareness—being cautious only when necessary.

Referentially ambiguous questions vs. non-ambiguous questions.

Methods

The authors introduce AMBISTEER, a framework that uses activation steering to modify a model's internal representations during the generation process without changing its underlying weights.

  1. Contrastive Pairs: The researchers created datasets of "minimal pairs" where the same image and question are paired with two different responses: one that is ambiguity-aware and one that commits to a single interpretation.
  2. Steering Vector Extraction: By calculating the difference in the model's internal hidden states (activations) between these pairs, they derived a "steering vector." This vector represents the mathematical direction in the model's latent space that corresponds to "being ambiguity-aware."
  3. Conditional Gating: To ensure the steering is only applied when needed, they trained a simple linear classifier (a probe) on the model's own activations. This classifier predicts whether a given image-question pair is ambiguous.
  4. Inference-Time Steering: During a live conversation, if the classifier detects ambiguity, "positive steering" is applied to push the model toward clarification or enumeration. If the input is clear, "negative steering" is applied to suppress those behaviors and ensure a direct answer.
Overview of the AMBISTEER framework.

Datasets

The study adapted two existing VQA benchmarks to create their contrastive training signals:

  • ClearVQA: A dataset focused on clarification-seeking behavior.
  • AQuA: A dataset focused on referent enumeration (listing multiple possible answers). The authors performed manual relabeling of these datasets to ensure high-quality binary ambiguity labels for training their classifier.

Key Insights and Ideas

  • The "Aha" Moment for Steering: Steering alone is relatively weak, and prompting alone is inconsistent. However, combining prompting and steering is highly effective. Steering acts as a "reinforcement" for the instructions provided in the prompt, helping the model follow them more reliably.
  • Linearity of Ambiguity: Probing experiments revealed that the distinction between "ambiguity-aware" and "direct" behavior is nearly perfectly linearly separable in the model's middle-to-late layers. This suggests that the concept of ambiguity is a high-level feature that the model's internal representations already track.
  • Models "Know" They're Confused: The fact that a simple linear classifier can detect ambiguity with high accuracy (up to 93.8% on AQuA) suggests that VLMs internally "know" when a question is underspecified; they just lack the pragmatic training to act on that knowledge by default.

Key Results

The researchers evaluated AMBISTEER across three state-of-the-art 8B-parameter models: Qwen3-VL, InternVL3.5, and Molmo2.

  • Behavioral Accuracy: AMBISTEER increased the proportion of inputs handled appropriately (clarifying when ambiguous, answering directly when clear) from a baseline of 59.0% to 77.0% on average.
  • Selective Control: While unconditional steering caused models to "hallucinate" ambiguity on clear inputs roughly 66% of the time, AMBISTEER's gated approach reduced this undesired behavior to just 17.0%.
  • Performance Gains: On ambiguous inputs specifically, the combination of prompting and steering reached target behavior rates as high as 98% for some models, far outperforming either method in isolation.
Clarification and enumeration rates on balanced test sets.

Limitations and Future Work

  • The Classifier Bottleneck: The overall system performance is heavily dependent on the ambiguity classifier. On the ClearVQA dataset, classifier errors were the primary source of failure.
  • Factual Grounding: While AMBISTEER helps models decide to enumerate options, it doesn't guarantee those options are factually correct. The authors noted cases where a model correctly identified ambiguity but then hallucinated details about the objects it was listing.
  • Scaling: The study focused on 8B parameter models. Whether these linear steering properties hold for much larger (e.g., 70B+) or much smaller models remains an open question.
  • Future Directions: The authors suggest that this lightweight paradigm could be extended to other forms of uncertainty, such as when a model lacks the external knowledge required to answer a question.

Summary updated

Abstract

Vision-language models (VLMs) often exhibit overconfidence, providing firm answers even when visual evidence is insufficient. A prominent case is referential ambiguity in Visual Question Answering (VQA), where multiple entities in an image are plausible referents of a question. Addressing such cases requires recognizing when ambiguity is present: ambiguous questions can be handled by asking for clarification or enumerating plausible referents, while unambiguous questions should receive a direct answer. Current VLMs struggle with this distinction, often defaulting to a single interpretation. To investigate whether activation steering can serve as an inference-time mechanism to elicit such behaviors, we introduce AmbiSteer, a lightweight framework to selectively elicit ambiguity-aware behavior. Across three VLMs, we find that steering alone has limited and model-dependent effects, while prompting is generally more effective. Combining prompting and steering substantially increases clarification and enumeration behaviors, but also elicits them on unambiguous cases. AmbiSteer addresses this using a linear classifier to predict input ambiguity and condition the steering direction accordingly, increasing the proportion of inputs handled appropriately from 59.0% to 77.0% on average across models. Our work sets the ground for lightweight approaches to reduce model overconfidence in uncertain multimodal settings.
BibTeX
@misc{riedler2026ambisteer,
  author = {Monica Riedler and Samuele Antonelli and Barbara Plank and Raffaella Bernardi},
  title = {{AmbiSteer: Selectively Eliciting Ambiguity-Aware Responses in Vision-Language Models}},
  year = {2026},
  eprint = {2611.17900},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17900},
}
Figure