Knowing When the Detector Missed: Coverage-Aware Confidence for Specialized Visual Grounding

Many systems that find things in images work in two stages. A frozen object detector first proposes a few dozen candidate boxes. A second model then reads a text query, such as "the left lower lobe opacity," and picks one of those boxes. This design has a hard ceiling: if the detector never proposed a box around the target, the second model cannot find it. It still picks a box, though, and it often does so with high confidence. The resulting confidence score blends two different questions. Is the target among the candidates at all? If it is, did we pick the right one? In natural photos the first question is usually answered yes, so the blend does little harm. In specialized imagery, such as chest X-rays, where generic detectors routinely miss subtle findings, the first question is often answered no. A single confidence score then hides exactly the failure that matters.

We propose to separate the two questions explicitly. We train a small "coverage" estimator that predicts whether the candidate pool contains the queried region. Its training labels come for free from ground-truth boxes. Its output multiplies the selector's posterior. We test one claim. On domain imagery, most of a grounding system's overconfidence comes from detector misses rather than selection mistakes. An estimator that targets coverage directly should therefore fix calibration and abstention in a way that rescaling the selector's scores cannot. A second, testable claim follows. Compact models specialized for a domain should be more overconfident on coverage misses than generalists are, mirroring the "trigger-happy" behavior documented for specialized tool-calling models.

Core Claim

Let $q$ be a query and $P$ the detector's proposal pool. Let $c(q)\in\{0,1\}$ indicate that some proposal in $P$ overlaps the target with intersection-over-union (box overlap) at least $\tau$. Let $s(q)$ be the selector's probability for its chosen box. The probability that the final box is correct then factorizes as

$$ \Pr(\text{correct}\mid q,P) = \Pr(c=1\mid q,P)\cdot \Pr(\text{right pick}\mid c=1,q,P). $$

A softmax over $P$ always sums to one, so a standard selector estimates only the second factor, implicitly assuming the first equals one. We estimate the first factor with a separate coverage estimator $\hat\pi(q,P)$. We report $\hat\pi\cdot\tilde s$ as the confidence, where $\tilde s$ is the selector posterior recalibrated only on queries the detector covered, and we abstain when $\hat\pi$ falls below a threshold. The estimator is a three-layer perceptron, the same lightweight design as the hidden-state probe in Uncertainty Calibration for Tool-Using Language Agents. It reads summary features of the query and the pool: the query embedding, the sorted detector objectness scores, the entropy of the selector's attention over proposals, the maximum query-to-region similarity, and the fraction of image area the pool covers. Its labels come from measuring overlap between proposals and annotated boxes. Training it requires no extra annotation, and it never touches the selector's weights.

Relation to Prior Work

Three papers motivate the gap. Work on frozen-detector pipelines such as SeqGROUND, LXMERT, and LLMs as Bridges shows that downstream reasoning cannot recover regions the detector missed, especially rare or fine-grained ones. Neural Baby Talk shows that models fall back on language priors when a region is missed, which is a mechanism for confident error. How Do Medical MLLMs Fail? shows that medical multimodal models spread their attention over background rather than lesions and introduces the VGMED benchmark. Its fix, VGRefine, masks attention at inference time and produces no uncertainty estimate.

Three papers sit closest to the method, and each differs from ours on a specific point. Flexible Visual Grounding adds a "pseudo region" to the Faster R-CNN pool so the model can reject unanswerable queries. It treats rejection as one more class inside the same softmax, uses only natural images, and never measures calibration. Confidence-aware Pseudo-label Learning filters noisy training pairs with a BLIP-derived matching score. That score is a fixed heuristic threshold, not a calibrated probability, and it is used during training rather than at prediction time. HyperClick calibrates verbalized click confidence against a 2D Gaussian reward using reinforcement learning, but it works end to end on clean GUI elements and has no proposal stage to fail. Our approach differs from all three in two ways. We give coverage failure its own estimator, separate from selection, and we evaluate in domains where detector recall is genuinely poor.

Calibration is the Bottleneck shows that over 90% of errors on unavailable tools come from specialized models calling the tool anyway. We borrow its diagnostic stance, which separates outcome accuracy from whether the model took the correct action. Its finding that inference-time probes help some models and hurt others warns us to report results per model rather than averaged. Finally, AgriPerceiver reports an ECE (expected calibration error, the average gap between confidence and accuracy) of 0.139 for a specialized compact model and proposes temperature scaling as a fix. Our decomposition tests whether that fix is enough.

Research Program

We build grounding pipelines on two detector types. The first is a generic Faster R-CNN trained on the Visual Genome categories. The second is the same detector fine-tuned on domain boxes. For each detector we train two selectors. One is a generalist, a cross-modal transformer trained on RefCOCO. The other is a compact specialist fine-tuned on medical grounding data. We sweep the number of proposals per image so detector recall varies in a controlled way.

The first study is diagnostic. For each pipeline we split test queries into "covered" and "missed" by measuring overlap between the pool and the ground-truth box. We then report the selector's mean confidence and ECE separately in each stratum. Our hypothesis predicts that confidence on missed queries sits close to confidence on covered ones, and that the gap is larger for specialists.

The second study trains the coverage estimator and the covered-only recalibration on a held-out calibration split. It then evaluates calibration, selective prediction, and abstention.

The third study asks whether a coverage estimator trained on natural images transfers to medical images. This bears on the disagreement between How Do Medical MLLMs Fail?, which finds that grounding attention heads transfer across domains, and Seeing the Trees for the Forest, which finds that high-norm background tokens break transfer. Our features include attention entropy, so heads that fail to transfer should show up as a drop in coverage-estimator accuracy.

Baselines

The naive reference point is the raw selector's maximum softmax probability, applied after forced selection. Every other method must beat it, and it shows how far confidence drifts on missed queries.

The first strong competitor is temperature scaling of the selector on the same calibration split. This is the fix AgriPerceiver proposes. It is the cheapest serious alternative.

The strongest existing method for absent targets is the pseudo-region approach of Flexible Visual Grounding, which we follow with temperature scaling. That paper reports 91.2% accuracy on unanswerable queries and 69.7% on answerable queries on its RefCOCO+ Pseudo Dataset, and 81.7% overall on SMD4FVG. We re-implement it on our detectors and data with the same selector backbone, training budget, and calibration split. On the natural benchmarks we must first match its numbers to show the re-implementation is faithful. Beating it means lower box-level ECE and a smaller area under the risk–coverage curve on the medical sets, with answerable-query accuracy within one point of the baseline.

The second strong competitor is a probe in the style of PROBECAL Uncertainty Calibration for Tool-Using Language Agents, which predicts overall correctness from the same features. That paper reports ECE falling from 0.690 to 0.035 on TabMWP. That result comes from a text task, and no reported result found on Lacuna for probe-based calibration of box grounding.

The key ablation compares this single-target probe with our two-factor version. Both get the same features, capacity, and labeled data, so any gain must come from supervising coverage separately. We also report an oracle that sets the coverage factor to its true value. This gives an upper bound on what the decomposition can deliver.

Why not just…?

Why not just temperature-scale the selector? A single temperature rescales all queries alike. It can lower average confidence, but it cannot lower confidence on missed queries more than on covered ones unless the selector's logits already tell the two apart. Our diagnostic study measures exactly this. If temperature scaling alone brings missed-query ECE to the level of our method, the decomposition is unnecessary and we will say so. The stratified ECE results settle the question.

Why not just add a pseudo region and train rejection? This is the real threat. A pseudo region learns "no target present" from synthetic negatives that pair a query with an unrelated image. A detector miss is a different event: the target is present but went unproposed. The pseudo-region model may never see that event during training unless misses appear in its training data. Its reported 69.7% accuracy on answerable queries also suggests that the extra class costs accuracy on answerable queries. We give both methods the same naturally occurring misses during training. If the pseudo region then matches us, the contribution shrinks to the evaluation protocol and the specialist diagnosis, and we will report it that way.

Why not just raise detector recall? We will, as one axis of the sweep. Recall never reaches 100% on subtle findings. More proposals also add distractors, which hurts selection. The proposal-count sweep shows whether calibration error on missed queries disappears as recall rises, or whether it stays concentrated on the remaining misses.

Why not drop the detector for an end-to-end specialist model? End-to-end models still fail to attend to lesions, as How Do Medical MLLMs Fail? shows. Agentic open-vocabulary pipelines such as Connecting the Dots reintroduce a proposal stage, adding a caption bottleneck on top of it. We include one end-to-end medical model with a correctness probe as a further reference. We do not claim the two-stage design is better, only that its confidence can be made honest.

Proposed Experiments

Our main data are MS-CXR, which has bounding boxes for 8 diseases, and VGMED, which has about 28K clinician-built triplets of image, box, and question. We take natural absent-finding negatives from RSNA Pneumonia images labeled negative, rather than relying only on random query–image swaps. RefCOCO+, its Pseudo variant, and SMD4FVG serve as natural-image sanity checks against the published pseudo-region numbers. A result counts as correct when the chosen box overlaps the target with IoU of at least 0.5.

We report four metrics: box-level ECE and Brier score, both computed separately for covered and missed queries; area under the risk–coverage curve; and abstention accuracy on absent findings. Two experiments are decisive. In the first, our method must cut missed-query ECE substantially below the best competitor's on MS-CXR and VGMED without hurting covered-query ECE. In the second, the two-factor estimator must beat the single-target probe when both receive identical inputs and training labels. If specialists turn out no more overconfident on misses than generalists, that is a clean negative result for the second claim.