Regenerable Panoptic Questions for Diagnosing Phrasing Brittleness

When a vision-language model answers "is the swimmer in the water?" correctly but fails on "does the person's body touch the lake surface?", the failure has at least three possible causes. The model may have memorized the original question about a public image. It may be sensitive to wording for its own sake. Or it may not see the swimmer clearly, and fall back on word cues when the visual evidence is weak. Current benchmarks cannot separate these causes. They are static, so their images and questions may have been crawled into pretraining data. Their wordings are fixed. And when a perception error and a reasoning error produce the same wrong answer, the benchmark cannot tell them apart. We propose to build questions directly from panoptic segmentations, meaning annotations where every pixel is labelled with a class and an instance, covering both countable objects ("things": boats, people) and amorphous regions ("stuff": water, sky, shoreline). Because each answer is computed from the masks, an unlimited number of fresh, correctly answered questions can be generated over any annotated image, in any number of verified phrasings.

This generator lets us test a specific claim. A large share of what looks like prompt sensitivity in grounded visual reasoning is perceptual uncertainty showing up through language. If the claim holds, giving the model explicit panoptic masks should shrink its variation across phrasings. The shrinkage should be largest for relations at boundaries between objects and stuff regions, such as a person touching water, where bounding boxes are least informative. We test this in a specialized domain where generalist models are weak (maritime obstacle scenes from LaRS) and in a general-domain control built on COCO-derived images (PSG). We also include a freshly captured image set that cannot be in any training corpus. Those fresh images separate memorization from brittleness.

Core Claim

Two measurable statements make up the claim. First, the accuracy drop a model shows when a question is reworded can be split into a memorization component and a brittleness component. The memorization component appears only on publicly crawlable images. The brittleness component persists on never-published images. Second, the brittleness component depends on perception. It is concentrated on items whose visual evidence the model finds ambiguous, and it falls when the model is conditioned on panoptic masks. If both statements hold, the usual remedy for prompt sensitivity, training on more paraphrases, treats a symptom. The more direct remedy is better perceptual grounding.

The decomposition is simple to write down. Let $A(s, p)$ be accuracy on image source $s$ (public or fresh) under phrasing regime $p$ (the canonical dataset wording, or held-out generated paraphrases). We define the phrasing gap on each source as $\Delta_s = A(s,\text{canon}) - A(s,\text{para})$. We then estimate

$$ M = \Delta_{\text{pub}} - \Delta_{\text{fresh}}, \qquad B = \Delta_{\text{fresh}}, $$

where $M$ is the part of the gap that exists only when the image may have been seen in training, and $B$ is the part that remains on unseen images. This subtraction is only meaningful if the two image sources are matched in difficulty. We control for this by matching items on mask area, predicate type, and object-versus-stuff composition, and by reporting each $\Delta$ within matched strata.

Research Program

The question generator. It takes a panoptic annotation and derives relations geometrically, with no relation labels required. Two masks "touch" if a slightly dilated copy of one overlaps the other. "On" means touching plus a vertical ordering of the masks. "Inside" means the subject's pixels are mostly contained in the object's region. Proximity and counting questions are computed from mask distances and instance identities. Restricting the predicate set to geometrically defined relations is deliberate. Domain-wise Invariant Learning for Panoptic Scene Graph Generation treats subject-object pairs as separate environments and applies invariant training to counter predicate ambiguity such as "sitting on" versus "on". Its results show that human predicate labels carry reporting bias. We avoid that bias by defining answers from geometry rather than from human predicate labels.

The design borrows its logic from DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks, which generates reasoning problems from random graphs and varies the order in which they are described. DyVal showed that models strong on static benchmarks can score near zero on freshly generated items. That work stays within text-only symbolic tasks. Here a panoptic scene graph plays the role of DyVal's graph, and the images are real.

Phrasings. Each relation type gets a family of templates written by people and by a language model. Annotators check a sample of each template for meaning preservation, in the spirit of the human verification used by STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving. The families vary three things: word choice (including domain names such as "hull" for boat), clause order (subject-first versus object-first, mirroring DyVal's order reversal), and distractor clauses. Templates are split by family into training and held-out sets, so that any paraphrase-augmentation baseline is tested on wordings it never saw.

Domain rules as a separate step. For the maritime set, some questions add a rule stated in the prompt. For example: a person counts as a hazard if their mask touches the water region and lies below the shoreline region. The answer is still computed from the masks. We also record, for every item, whether the model got the perception sub-question right. This splits each failure into perception, rule application, or wording, the split that A Study of Situational Reasoning for Traffic Understanding argues existing benchmarks conflate. That study gets the split by removing vision entirely; we keep vision and gain it through construction.

Image sources. PSG images come from the overlap of COCO and Visual Genome, with 133 classes. They are almost certainly crawled. We use LaRS's 4,006 maritime keyframes, with 3 stuff and 8 thing classes; LaRS is public but domain-specific and less likely to have been heavily crawled. The third source is a new set of roughly 500 maritime frames, captured after the training cutoffs of the evaluated models, panoptically annotated in the LaRS label space, and never released. This fresh set is the part that cannot be faked, and its small size is why we stratify carefully.

Baselines

The naive reference point is the base vision-language model, LLaVA-1.5 at 13B, evaluated on canonical wordings only. This is what a static benchmark would report. Every other number is read against it.

The strongest existing method for our hypothesis is VCoder, from VCoder: Versatile Vision Encoders for Multimodal Large Language Models. VCoder feeds a segmentation map through a frozen image encoder with a trainable projector into LLaVA-1.5. It is the most direct existing way to give a language model panoptic masks. Its paper reports a Count-Score of 87.2 on COST, against 35.8 for base LLaVA-1.5 and 38.4 for GPT-4V. It also reports that the same paper names handling of wording variation as an open limitation, which is exactly what we test. We run VCoder under two conditions. In the oracle condition it receives ground-truth panoptic masks. In the realistic condition it receives predicted masks from Mask2Former, which LaRS: A Diverse Panoptic Maritime Obstacle Detection Dataset and Benchmark reports as the top panoptic model on that dataset, with a panoptic quality of 41.7%. The comparison is fair because all models answer identical generated items, receive the same image resolution, and use the same answer parser.

The competing remedy is supervised fine-tuning of base LLaVA-1.5 on generator questions using the training template families. It gets the same number of gradient steps and the same hyperparameter search budget as the mask-conditioned model, but no masks. We found no reported result on Lacuna for paraphrase-augmented fine-tuning on mask-derived relation questions. Prompt-Robust Vision-Language Models via Meta-Finetuning, which reports a 1.13-point average gain in harmonic mean accuracy on CLIP classification, works only with classification prompts. It is not directly applicable, so we do not claim to beat it.

The key ablation replaces masks with boxes. We feed VCoder-style rendered bounding boxes derived from the same masks, with matched training. This isolates whether pixel-level stuff boundaries matter, or whether any explicit localization signal would do.

The mask-conditioned model "beats" the paraphrase-trained model if it meets two conditions. On held-out wordings over fresh images, it must reduce $B$ by more than the paraphrase-trained model does. And the confidence intervals must separate, using a bootstrap over images.

Why not just…?

Why not just paraphrase existing benchmark questions with a language model, as PromptBench does? PromptBench: A Unified Library for Evaluation of Large Language Models perturbs static items at the character, word, sentence, and meaning level. That measures wording sensitivity, but it leaves the image-answer pairs unchanged. The same memorized pair can therefore survive a rewording, or fail because the paraphrase quietly changed the meaning. Either way, $M$ and $B$ cannot be separated. We run this approach as a measurement baseline on PSG. If its sensitivity estimates agree with ours on fresh images, the generator is not needed for measuring brittleness alone. It would still be needed for the memorization split and the perception split.

Why not just evaluate on a private held-out image set and skip the generator? This is a real threat, and our design includes such a set. Fresh images alone do remove contamination. What they lack is ground-truth relation questions in controlled wordings, unless someone writes them by hand. They also give no matched public counterpart for estimating $M$. The generator makes the fresh set cheap to query densely: dozens of verified items per image instead of one or two hand-written ones. That density is what makes stratified estimates feasible on about 500 frames.

Why not just use boxes or 3D coordinates? STSBench shows that even a text-only model given ground-truth trajectories reaches only 57.08%, with a drop of 20 to 30 points on agent-to-agent interactions. That suggests box-level geometry is not enough for interaction reasoning. Boxes also cannot express contact with water or shoreline. The box ablation above settles the question directly. If boxes reduce $B$ as much as masks do on contact relations involving stuff regions, our pixel-level argument fails.

Why not just fine-tune on more paraphrases? Instruction-tuning results such as Finetuned Language Models are Zero-Shot Learners show that wording diversity in training helps zero-shot transfer. However, DyVal's 13.6% drop for GPT-4 under order reversal shows that large-scale tuning does not remove structural sensitivity. The paraphrase-trained baseline tests this directly. If it matches mask conditioning on held-out wordings, the perceptual explanation is wrong, and we will report that.

Proposed Experiments

The first decisive check is the decomposition. We expect $M > 0$ on PSG and $M \approx 0$ on LaRS-versus-fresh comparisons. If $M$ is large even within the maritime domain, LaRS itself has leaked into training.

The second check is the central test. On fresh images, we compare $B$ across base, box-conditioned, oracle-mask, and predicted-mask models, broken down by relation type (thing-thing versus thing-stuff contact). We also report consistency, defined as the fraction of items answered identically across all held-out wordings. The claim is supported if oracle masks cut $B$ substantially more on thing-stuff contact items than on thing-thing items, and more than paraphrase fine-tuning does. The claim is falsified if $B$ is flat across conditions.

The third check correlates per-item brittleness with perceptual difficulty. Difficulty is measured both by the model's own accuracy on the perception sub-question and by the predicted-mask quality on that item. LaRS reports weak recognition of small objects such as swimmers and buoys, so we expect brittleness to concentrate there.

The fourth check uses the rule-step items to report how often failures come from perception, from rule application, or from wording, with and without masks. This tests whether masks fix the safety-relevant step that Multimodal Situational Safety found models skip even when they describe a hazard correctly.

All results are reported with bootstrap intervals over images and with the generator's human-verified error rate subtracted out as label noise.