Vision-language models can now answer questions and draw masks at the same time. Ask one whether the road from the bridge reaches the harbor, or whether two vessels in a retinal image join, and it will give a fluent yes or no. Nobody has checked whether that answer comes from seeing the structure or from guessing. Connectivity is a hard test of seeing. Erasing a three-pixel strip of road changes almost nothing in the image and almost nothing in a pixel-overlap score such as Dice, yet it changes the true answer from "connected" to "not connected." We propose to build matched image pairs that differ only by such a tiny cut or bridge. With these pairs we can measure topological hallucination: cases where a model's answer about connectivity, or its own mask, ignores a change in the structure's shape that is plainly visible. We then test whether training on these pairs, with losses that score the shape of the structure rather than its pixel overlap, makes the model's words and masks follow the actual topology.
The core claim is narrow and testable. Current pixel-grounded models answer connectivity questions from scene priors ("roads usually connect"), not from the structure in the image. Their masks are trained only to maximize overlap, so they cannot carry the needed signal, because a one-pixel gap is invisible to Dice. Pairing gap-flip counterfactuals with a topology-aware mask loss, and requiring the text answer to agree with the connectivity of the model's own mask, should fix both. The new pieces are the evaluation, which isolates topology from overlap, and the finding about which ingredient matters. Attaching a known topology loss to a known model is not the contribution.
Where this sits in prior work
Two lines of work motivate the problem. HalluSegBench built factual/counterfactual image pairs in which an object is replaced by a similar distractor. It showed that pixel-grounded models such as LISA and GLaMM draw masks for objects that are absent, and its counterfactual fine-tuning (RobustSeg) cut its Confusion Mask Score by 30%. Its edits swap whole objects and its metrics are pure overlap, so a mask that is correct except for one severed branch scores as nearly perfect. We borrow its paired design and change what the pair varies: the topology of a single structure rather than the identity of an object. Tri-HE showed that relation hallucination is worse than object hallucination, with LLaVA-1.5 at 37–40%. It scores relations as symbolic triplets checked against scene graphs. "A connects to B along this vessel" is a relation that no scene graph records and that only pixels can verify. POPE showed that adversarial negatives drawn from co-occurrence priors expose models: MiniGPT-4 F1 drops from 78.86 under random sampling to 71.37 under adversarial sampling. Gap-flip pairs are the topological version of adversarial negatives, since the prior says "connected" and the pixels say otherwise.
The pixel-grounded models we test are LISA and GLaMM. LISA passes a special token's embedding from a language model into a Segment Anything (SAM) mask decoder. GLaMM does the same while interleaving masks with dialogue. Both train their masks with binary cross-entropy plus Dice, which is exactly the objective the topology literature identifies as blind to breaks.
Two papers supply the method. clDice compares soft, differentiable skeletons of the prediction and the ground truth, which makes the loss sensitive to breaks at under 10% overhead. TopoMatch matches persistent-homology features (connected components and loops that persist across probability thresholds) only when they overlap in space. It showed that unconstrained matching pairs features in the wrong places up to 99.4% of the time. Its cubic cost in pixel count is the main constraint on our design, so we apply it to cropped, downsampled regions around the queried structure rather than to full SAM-resolution masks.
Two papers constrain the design. HALC reduces object hallucination by zooming into detector crops. For connectivity this may backfire, because a crop can cut the very path in question. We test this directly rather than assume it. TOHA detects hallucination from the topology of attention graphs in text-only models. We do not adopt it. Our hypothesis is that the decisive topology is in the image, and attention-graph topology has no established link to mask connectivity.
Research Program
The first component is a benchmark, which we call GapFlip. It draws on datasets whose masks already encode topology: DRIVE, Massachusetts Roads, and CREMI from the clDice and TopoMatch suites, TorontoCity from DeepRoadMapper, and STARE and CHASE-DB1 from the tubular-structure review.
The benchmark has three question types. The first type uses no image editing at all. From the ground-truth mask we sample two marked points that lie either on the same connected component or on different ones, and ask whether they are joined. The second type uses edited pairs. For each factual image we inpaint a minimal cut, or a minimal bridge, and apply the same edit to the ground-truth mask. The edit is kept under 1% of the structure's pixels and must change the number of components or loops. The third type asks for counts of separate pieces and enclosed loops, which correspond to the 0-dimensional and 1-dimensional homology of the mask.
We also check that the edits are visible. A human audit on a sample confirms that people can see the edit, which ensures the benchmark measures model failure rather than impossible perception.
The metrics come in three groups. Answer accuracy is reported separately on factual and counterfactual members of each pair. Flip consistency is the fraction of pairs where the answer changes when the truth changes. Mask quality is reported as Betti error and clDice alongside Dice, so that cases with high Dice and broken topology are visible. Answer–mask agreement checks whether the model's text matches the connectivity of its own mask, computed with union-find.
The second component is a diagnosis. We run LISA and GLaMM zero-shot on GapFlip, along with general vision-language models that answer in text only. We ask whether answers track the image at all. One test is whether the "yes" rate stays fixed across factual and counterfactual pairs, the bias POPE observed. The other is whether text errors line up with mask breaks, or occur even when the mask is correct, which would mean the language side ignores the mask.
The third component is training. We fine-tune LISA-7B on GapFlip training splits, built from training images only, using three additions. The first is a topology-aware mask loss that adds soft-clDice everywhere and TopoMatch on 128×128 crops around the queried path. The second is a paired loss that requires the factual and counterfactual masks to differ in Betti numbers where the truth does. The third is an agreement term that supervises the text answer with the connectivity computed from the ground-truth mask and penalizes disagreement with the model's own thresholded mask. At inference, an optional verified mode reads the answer directly from the model's own mask topology.
Baselines
The naive reference is LISA-7B and GLaMM off the shelf. As anchors for what these models do on their home tasks, LISA-13B reports 65.0 gIoU on ReasonSeg and GLaMM reports 79.5 mIoU on refCOCO val. No reported result was found on Lacuna for any vision-language model on connectivity questions, so these zero-shot numbers will be new.
The strongest existing method for hallucinated masks is RobustSeg-style counterfactual fine-tuning from HalluSegBench. It reports a 30% reduction in Confusion Mask Score and 83.37% localization accuracy on FP-RefCOCO. We reimplement its positive-negative loss on GapFlip pairs with the same LISA-7B starting point, the same data, the same number of steps, and the same hyperparameter search budget, but keep its overlap-only mask loss. Beating it means higher flip consistency and lower Betti error at equal or better Dice on held-out structures.
The strongest decoding-time competitor is HALC, run with its released code (BillChan226/HALC) on the text-answer task. It reports cutting MiniGPT-4 sentence hallucination from 30.87% to 17.80% on CHAIR. A training-free comparison is RITUAL, which reports raising LLaVA-1.5 POPE accuracy from 84.13% to 88.87%.
The specialist reference is a pipeline: a UNet trained with clDice on the same structures, followed by union-find on the predicted mask. For roads we also use the DeepRoadMapper pipeline, which reports 84.0% graph F1 on TorontoCity. This is an upper reference for in-domain structures, not a competitor we expect to beat there.
Two ablations isolate the key ingredient. The first adds clDice and TopoMatch to the mask loss but drops the paired loss and the agreement term. The second keeps the pairs and the agreement term but uses Dice only.
Why not just…?
Why not just add soft-clDice to LISA's mask loss and stop? This is the obvious combination, and it is our first ablation. Our expectation is that it repairs masks but not answers, because nothing ties the language output to mask connectivity. HalluSegBench's finding that priors persist after fine-tuning points the same way. If this ablation matches the full model on flip consistency, the pairing and agreement terms are unnecessary and we will report that.
Why not just call a specialist segmenter and a graph algorithm as a tool? On DRIVE-like or road-like images with trained specialists, this is a real threat and will probably win in-domain. It needs a specialist for each structure type, though, and DeepRoadMapper's multi-stage pipeline shows how much engineering it takes for even one domain. The decisive test is a held-out structure split of cracks, rivers, cables, and neuronal membranes, where no specialist was trained. If the tool still wins there, the conclusion is that generalist models should route connectivity through explicit topology computation rather than learn it, and the verified-mode result will show whether the model's own mask is good enough to serve as that tool.
Why not just morphologically close the mask, or zoom in as HALC does? Closing bridges true gaps as easily as false ones, so on cut counterfactuals it should increase false "connected" answers. That makes it a measurable failure on half of every pair, not a side effect we have to argue about. Zooming risks cropping out the path being asked about. We report HALC's accuracy as a function of how much of the true path its crop contains.
Why not just prompt the model to trace the path step by step, like Tri-HE's Triplet Description? That prompting reduced MiniGPT-4's hallucination rate only from 47.44% to 42.96% on symbolic relations. We include tracing prompts as a zero-cost baseline, and expect them to inherit the same prior bias on gap-flip pairs.
Proposed Experiments and Decisive Checks
The first decisive check is whether topological hallucination exists. If zero-shot LISA and GLaMM show near-equal "yes" rates on factual and counterfactual pairs while their masks keep Dice above 0.9 and Betti errors are high, the phenomenon is real and invisible to current metrics.
The second check asks which ingredient fixes it. We compare the full model against the clDice-only ablation and against RobustSeg-style fine-tuning on flip consistency and Betti error, using in-domain and held-out structure splits with three seeds each. The claim survives only if the full model beats both on the held-out split.
The third check is whether answers come from the mask. Answer–mask agreement should rise sharply under the agreement term. Verified mode should match or beat the free-text answers. If it does not, the language pathway is still bypassing the mask.
Two further checks guard against side effects. Scores on ReasonSeg gIoU and POPE must not fall by more than one point after training, so that topology training does not damage general grounding. For cost, TopoMatch on 128×128 crops should add training time within the same order as clDice's under-10% overhead, and we will report wall-clock time for both.
References
- HalluSegBench: Counterfactual Visual Reasoning for Segmentation Hallucination Evaluation
- Unified Triplet-Level Hallucination Evaluation for Large Vision-Language Models
- Evaluating Object Hallucination in Large Vision-Language Models
- LISA: Reasoning Segmentation via Large Language Model
- GLaMM: Pixel Grounding Large Multimodal Model
- clDice - A Novel Topology-Preserving Loss Function for Tubular Structure Segmentation
- Topologically Faithful Image Segmentation via Induced Matching of Persistence Barcodes
- HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding
- Hallucination Detection in LLMs via Topological Divergence on Attention Graphs
- DeepRoadMapper: Extracting Road Topology from Aerial Images
- Human treelike tubular structure segmentation: A comprehensive review and future perspectives
- RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in LVLMs