Recent segmentation systems work like a person correcting a drawing. A multimodal language model looks at the image and its current mask, writes a short critique ("the mask spills onto the table edge; the left handle is missing"), and then places corrective clicks for a promptable segmenter such as SAM, the Segment Anything Model. It repeats this loop until it decides the mask is good enough. These agents are trained with reinforcement learning on final mask overlap. Work on reasoning faithfulness in language models suggests that this kind of training produces "reasoning shells": text that sounds like the cause of the answer but is not. Nobody has checked whether the critique in a segmentation loop actually drives where the clicks land. Nobody has checked whether the agent even looks at its own previous mask, rather than re-segmenting from the query each round.
This note proposes two things. The first is a measurement: counterfactual interventions on the two inputs of each refinement step, the current mask and the critique text, to see whether either one causally governs the next action. The second is a training recipe that makes the critique load-bearing. It uses deliberately corrupted masks, whose errors we know, as verifiable targets for the critique. A sufficiency reward then requires that a separate executor, seeing only the image and the critique, can reproduce the agent's correction. The practical payoff is error recovery as well as interpretability. Current agents cannot undo a wrong initial box. A critique that faithfully reports "this is the wrong object" is the natural channel for a reset action.
Core Claim
In iterative reasoning segmentation, we claim that the intermediate mask and the verbal critique are largely bypassed under terminal-reward training. We further claim that training the critique against known injected errors, and requiring it to be sufficient for the correction, will make it causally load-bearing. That should improve recovery from catastrophic early mistakes without lowering final mask quality. Both halves can be tested. If the critique turns out to be already faithful, or if dropping the text entirely loses nothing, the experiments below will show it.
Where This Sits
The motivating evidence comes from language-model faithfulness. RFEval forces counterfactual reasoning prefixes into large reasoning models and finds 49.7% of outputs unfaithful, with models "silently correcting" injected flawed premises. Measuring the Faithfulness of Thinking Drafts in Large Reasoning Models finds draft-to-answer consistency often below 30% on hard tasks. Both papers link accuracy-driven reinforcement learning to this loss of faithfulness. On the segmentation side, SAM-Veteran is the target system. It trains an agent with group relative policy optimization (GRPO, a reinforcement learning method that scores each sample against others drawn for the same prompt) to output a box, then corrective points with textual reasoning, then a stop decision. It reaches 68.2 gIoU on ReasonSeg. It never tests whether the reasoning steers the points, and it cannot revoke a bad box. SegLLM supplies the strongest hint that loops are bypassed: its authors report success even when mask-memory embeddings were withheld, because the model leaned on dialogue text and words like "left." FineRS shows the visual version of post-hoc rationalization, where the thinking trace describes the target correctly while the chosen crop misses it.
The closest method is On the Faithfulness of Visual Thinking (SCCM). It measures treatment effects of corrupting intermediate visual crops in visual question answering. It then trains with a sufficiency reward, where a judge must answer from the crop alone, and a minimality reward. We reuse its sufficiency idea but change three things. The intermediate object is a pixel mask inside a multi-step loop, not a bounding-box crop for a question. The thing made sufficient is the critique text, which must be sufficient for the next correction rather than for an answer. The interventions target the agent's own previous output.
FRIT supplies the second method ingredient: building preference pairs from interventions on reasoning steps. Markovian Transformers supply the bottleneck idea, an answer module that cannot see the prompt. We use that idea both inside the sufficiency reward and as a competing baseline.
SAM-Veteran's own auxiliary task, locating the "centers of flaws" in masks corrupted with random blobs, already trains a related skill. We differ in using corruptions as causal interventions during refinement trajectories. We also add semantic corruptions (wrong object, wrong part, shifted region), not only random blobs.
Two negative results constrain the design. Measuring and Mitigating Post-Hoc Rationalization in Reverse Chain-of-Thought Generation shows that telling a model not to anchor makes anchoring worse. It finds that abstract structural tags help instead, so we give the critique a fixed slot structure rather than instructions. Dismantling Pathological Shortcuts shows that unfaithful outputs often have the same visual attention volume as faithful ones, which rules out "attend harder to the mask" as a fix.
Research Program
Phase 1: audit. We define two per-step interventions on a trained agent. In the mask intervention, the agent's current mask is replaced by a corrupted version with a known error: an added blob, a removed part, the mask of a different same-class object, or a translated region. We then record whether the critique and the next points change in the direction the corruption demands. In the critique intervention, the generated critique is replaced, by prefix forcing in the style of RFEval, with a well-formed critique describing a different error. We then record whether the points follow the new text or the original visual evidence. We summarize the second as a follow rate:
$$F = \Pr\big[\, d(p', \tilde r) < d(p', r) \,\big],$$
Here $r$ is the original critique's error region, $\tilde r$ the substituted critique's region, $p'$ the points placed after substitution, and $d$ the distance from points to region. $F$ near 1 means the text steers the clicks. $F$ near 0 means the clicks ignore the text. We also report the treatment effect of the mask intervention on next-step IoU change. Phase 1 runs on SAM-Veteran, or a faithful retrain if weights are unavailable, and on GRPO checkpoints across training. That tests whether the reasoning-shell effect seen in language models appears in segmentation as reward optimization proceeds.
Phase 2: training. The critique gets a fixed structure with slots for error type (spurious region, missing region, wrong object), location, and action, where the action can be add, remove, or reset, which discards the current box and regrounds. During GRPO, a fraction of trajectories start from an injected corrupted mask. Because the corruption is known, the critique's error-type and location slots can be scored exactly. That gives a verifiable reward for honest critique, the way final IoU is verifiable for masks. A second, sufficiency reward runs a frozen executor that sees only the image and the critique, never the policy's hidden state or the query. The policy is rewarded when the executor's correction attains IoU gain close to the policy's own. Following FRIT, critique-swap pairs in which the policy's points ignored a substituted critique become negative preference examples. To address the occlusion problem SAM-Veteran reports, masks are fed back as a contour or a separate channel rather than a filled overlay. This is held fixed across all systems so it does not confound comparisons.
Baselines
The naive reference points are one-shot segmentation and a single-step version of the agent. LISA reports Grounded-SAM at around 26% gIoU on ReasonSeg (LISA). We will also run our own agent truncated to its first box plus one round, which shows how much iteration contributes at all.
The strongest existing method is SAM-Veteran, with a reported 68.2 gIoU on ReasonSeg and 2.08 average steps (SAM-Veteran). For FINERS-4k, the strongest number in the file is FineRS at 55.1% gIoU (FineRS). To keep the comparison fair, we retrain SAM-Veteran's recipe on the same backbone, training data, GRPO sample budget, and mask-feedback rendering as our method, and we also report its published number. Beating it means matching ReasonSeg gIoU within noise (roughly ±1 point across seeds) while raising the critique follow rate $F$, the mask-intervention effect, and the recovery rate from injected wrong initial boxes. A faithfulness gain that costs more than about two gIoU points would count as a trade-off, not a win.
Three ablations isolate the key ingredient. The first is identical GRPO with corrupted-start trajectories but only the IoU reward, which separates "seeing corrupted masks" from "being rewarded for describing them." The second removes the sufficiency reward. The third removes the reset action, which separates recovery gained through faithful critique from recovery gained merely by having a reset button. We also run a selection-gate baseline that accepts a refinement only if a quality estimate improves, in the spirit of SAM-REF. It addresses round-to-round regressions with no faithfulness component at all.
Why not just…?
Why not just drop the text and train a point-only policy? If the critique is decorative, deleting it may cost nothing in IoU, and this is the most serious threat to the proposal. We train a point-only agent, with reset available, under the same budget. If it matches our method on gIoU and on recovery from wrong initial boxes, the honest conclusion is that verbal critique adds only monitorability, and we will report it that way. Our hypothesis is that recovery from wrong-object errors needs an explicit, discrete judgment that the whole region is wrong. FineRS's unrecoverable crops suggest that local click policies do not form this judgment on their own.
Why not just impose a Markovian bottleneck, where the refinement step sees only the image and the current mask, not the query or history? This directly blocks the text-history shortcut SegLLM observed, and in language Markovian Transformers lift GSM8K from 19.6% to 57.1% with such a constraint. But hiding the query makes the agent unable to detect wrong-object errors, which require knowing what was asked. We run it as a baseline: it should score well on the mask-intervention effect and poorly on wrong-object recovery.
Why not just instruct the model to base its clicks on its critique? Measuring and Mitigating Post-Hoc Rationalization in Reverse Chain-of-Thought Generation finds such suppression instructions increase anchoring. We include a prompted variant only as a cheap check in the Phase 1 audit.
Why not just scale the backbone? RFEval found a 120B model less faithful than its 20B sibling, so scale is not expected to fix bypassing. Where compute allows, we audit two backbone sizes to see whether this holds in segmentation.
Proposed Experiments
Accuracy is measured on ReasonSeg (gIoU and cIoU), RefCOCOg (cIoU), and FINERS-4k (gIoU by size bin), where small objects make early crop errors common. Faithfulness is measured on a held-out intervention suite built from ReasonSeg and RefCOCOg validation images. The suite pairs each image with corrupted masks of each error type and substituted critiques, and reports the follow rate $F$, the mask-intervention effect, and the wrong-box recovery rate. Recovery is the fraction of trajectories started from a box on a distractor object that end with IoU above 0.5. No reported result was found on Lacuna for recovery rates of SAM-Veteran or FineRS under injected initial errors, so these numbers are new. FlowSeg's rescue of 44.6% of its own cIoU<0.5 cases (FlowSeg) is a loose reference, not a like-for-like comparison. General multimodal benchmarks used by SAM-Veteran (MMStar, MME) check for forgetting.
Three outcomes are decisive. First, if Phase 1 finds high $F$ and a large mask-intervention effect in SAM-Veteran already, the motivating premise fails and the paper becomes a positive audit. Second, if the IoU-only corrupted-start ablation matches the full method on recovery, then the faithfulness rewards are not what drives the gain. Third, if the point-only policy matches on recovery, the critique is not needed for performance. Finally, tracking $F$ across GRPO checkpoints answers, for segmentation, whether terminal-reward training erodes faithfulness as it does in language reasoning models.
References
- RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Models
- Measuring the Faithfulness of Thinking Drafts in Large Reasoning Models
- SAM-Veteran: An MLLM-Based Human-like SAM Agent for Reasoning Segmentation
- SegLLM: Multi-round Reasoning Segmentation with Large Language Models
- FineRS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learning
- On the Faithfulness of Visual Thinking: Measurement and Enhancement
- FRIT: Using Causal Importance to Improve Chain-of-Thought Faithfulness
- Markovian Transformers for Informative Language Modeling
- Measuring and Mitigating Post-Hoc Rationalization in Reverse Chain-of-Thought Generation
- Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding
- LISA: Reasoning Segmentation via Large Language Model
- SAM-REF: Introducing Image-Prompt Synergy during Interaction for Detail Enhancement in the Segment Anything Model
- FlowSeg: Dynamic Semantic Guidance for LLM-Conditioned Segmentation