When more than half of an image is missing, inpainting models (generators that fill in the missing region) tend to fail in a recognizable way. Roads, wall edges, cell membranes and vessels that enter the hole from the visible border come out broken: they stop partway in, or they reappear as disconnected stubs. Pixel scores barely register this, because a missing one-pixel bridge is almost nothing by area. Functionally, though, it destroys the result. A severed road network is no longer a road network.
This note argues that the failure has a specific internal cause and a specific cure. The cause is that transformer attention fades with depth into the hole. Image positions far from the visible border pay less and less attention to the few border positions where a line actually enters, much as vision-language models stop attending to the image as their text output grows longer. The cure is a connectivity penalty computed from persistent homology (a way of counting connected pieces and loops in a soft probability map while tracking which pixels create or break them). The penalty is anchored on those border entry points. Because it needs only the visible context, it can be applied both during training and as a refinement step at test time, when the hole's true content is unknown.
Core Claim
Two-stage inpainters such as ZITS++ work in two steps. They first predict a line sketch of the missing structure using an axial-attention "structure restorer" at 256×256 resolution. They then paint texture on top of that sketch. The structure restorer is trained with pixel, cross-entropy and adversarial losses, and its authors report that it still breaks down on very large masks and dense junctions. We make two claims.
The first is diagnostic. Inside the hole, the places where the predicted sketch fragments coincide with the places where attention from hole positions to border entry points has decayed. Fragmentation sites can be located precisely as the saddle pixels that persistent homology identifies, and their depth into the hole should predict the error better than mask ratio alone does.
The second is about the remedy. A topological loss defined relative to the hole boundary repairs fragmentation that full-image topological losses do not reach. This loss penalizes floating islands and leaves line entries unmatched. It also works without ground truth, so it can fine-tune the sketch on each test image.
That second property is what separates this proposal from simply bolting an existing topological loss onto an inpainter. TopoMatch compares a prediction to a ground truth and matches features only where they physically overlap. That requires the full ground-truth mask, so it cannot be applied at inference inside a hole.
Relation to Prior Work
Three works supply the method:
- Topology-Preserving Deep Image Segmentation introduced persistence-diagram losses that push gradients onto the critical pixels creating spurious components or gaps. It also handled cost by computing homology relative to the borders of 65×65 patches. That assumption degenerates when a patch lies entirely inside a hole with no observed border, and our boundary-anchored variant is designed for exactly that case.
- TopoMatch fixed the spatial blindness of plain Wasserstein matching. The file reports that plain matching pairs features with the wrong location up to 99.4% of the time. We reuse TopoMatch as our training-time supervised loss and as our strongest baseline.
- TopoGAN showed that a persistence loss on generated masks prevents shattered roads and membranes, lowering a Betti score on CREMI from 0.236 to 0.015. It works on unconditional mask synthesis, not on completion conditioned on a partial view.
Prior work on attention decay supplies the diagnostic lens. iTaD ties hallucination in vision-language models to falling attention on image tokens. Its authors concede it tracks only total attention mass and is spatially blind. Vista-llama attributes decay to positional distance and equalizes it. RAR and More Thinking, Less Seeing? show late layers dumping attention onto uninformative tokens.
We carry the spatial version of this question into vision. The question is not whether attention reaches the visible region at all. It is whether attention reaches the few visible pixels that carry the line. Segmentation provides a related clue. Lv et al. found that a Swin-UNetR trained with clDice still fails at the low-contrast ends of the mandibular canal. We read this as the same anchor-starvation effect, with low contrast standing in for missing pixels.
Research Program
The first component is measurement. Take the structure restorer of ZITS++ and define, for each hole position $p$ and layer $\ell$, an anchor-attention score $a_\ell(p)$. This is the attention mass that $p$ places on visible border tokens lying on predicted or observed line pixels, normalized by the mass $p$ places on all visible tokens.
Plot $a_\ell(p)$ against distance into the hole. Separately, run 0-dimensional and 1-dimensional persistence (counting connected pieces and closed loops, respectively) on the predicted sketch versus the ground truth inside the hole, and extract the saddle pixels responsible for wrong merges and missed merges. The diagnostic hypothesis is that error saddles are concentrated where $a_\ell$ has fallen below a threshold. If error saddles turn out to be spread uniformly, or tied only to junction density, the attention story is false and the attention intervention below is dropped.
The second component is the boundary-anchored loss. Let $E$ be the predicted edge likelihood map, $M$ the hole, and $B$ a thin ring of visible pixels around it. Let $\mathcal{K}$ be the set of line entries, meaning the connected runs of observed line pixels on $B$.
We run a superlevel-set filtration of $E$ on $M \cup B$. In this filtration we sweep a threshold down from 1 and track when pieces of the map appear and merge. Entry pixels are held at likelihood 1, but each entry is kept as its own seed rather than merged in advance. Each component of the prediction is then one of two kinds. It is either an island, born inside $M$ at a peak of likelihood $b_c$ and merged into the rest at likelihood $d_c$. Or it is an entry, whose first merge with another entry happens at saddle value $s_k$.
The loss is
$$ \mathcal{L}_{\text{anchor}}(E) = \sum_{c \in \mathcal{I}} (b_c - d_c)^2 \;+\; \sum_{k \in \mathcal{K}} w_k \,\big(\tau - s_k\big)_+^2 , $$
with the terms defined as follows:
- $\mathcal{I}$ is the set of islands.
- $\tau$ is a target merge likelihood.
- $(\cdot)_+$ means the positive part.
- $w_k \in [0,1]$ is the predicted probability that entry $k$ should connect to some other entry rather than terminate.
The first term shrinks floating fragments. The second term raises the weakest pixel on the best path joining an entry to its partner. As in TopoNet, gradients land only on critical pixels. The weights $w_k$ come from a small head trained on ground truth to predict whether each entry continues. This matters because real lines do end inside holes. Without the weights, the loss would force spurious bridges, and that is the main risk of the design.
During training, $\mathcal{L}_{\text{anchor}}$ is added alongside TopoMatch on the hole region. At test time, TopoMatch is unavailable. We instead take a few gradient steps on the sketch logits, or on a low-rank adapter of the structure restorer, under $\mathcal{L}_{\text{anchor}}$ plus a term keeping the sketch close to its initial prediction. The refined sketch then passes unchanged to the texture stage.
On cost, persistence runs only on $M \cup B$ at 256×256 in 2D using union-find. This is the regime where the file notes that 2D computation is tractable.
The third component is an optional attention intervention, tested only if the diagnosis holds. It adds a distance-equalized positional bias from hole queries to entry tokens, in the spirit of Vista-llama. This lets us ask whether fixing attention alone recovers what the loss recovers.
Baselines
The naive reference point is the released ZITS++ structure restorer and texture network, run unchanged on our masks. On Indoor it reports PSNR 26.15 dB, against 25.57 dB for ZITS and 25.20 dB for LaMa (ZITS++). That is the only inpainting number in the file. We found no reported result on Lacuna for topological metrics of any inpainting method, nor for road or membrane inpainting under large masks. Every topological comparison will therefore be measured by us, under one protocol.
The strongest existing method is the ZITS++ structure restorer fine-tuned with TopoMatch Betti matching against the full ground-truth sketch. It is the strongest competitor because it is the best spatially correct topological loss available, and with ground truth present at training it could plausibly learn everything our loss teaches. A second topological baseline fine-tunes the same restorer with soft clDice, the skeleton-overlap loss used by Lv et al..
Fairness is enforced as follows:
- Every variant starts from the same checkpoint.
- Every variant sees the same images and masks for the same number of steps.
- Every variant gets the same sweep of eight loss-weight values. This matters because the file notes that topological losses are sensitive to their weight.
- Our test-time refinement adds compute, so we also give the baselines equal extra test-time compute: test-time augmentation averaging of the sketch.
Beating TopoMatch means two things on held-out masks above 50% area. First, a lower Betti-number error inside the hole and a higher fraction of correctly connected entry pairs, with non-overlapping confidence intervals across three seeds. Second, PSNR no more than 0.1 dB below the naive model.
Two ablations isolate the key ingredient. In the first, test-time refinement is removed, which tests whether the ground-truth-free property matters. In the second, the weights $w_k$ are set to 1, which tests whether the continuation head prevents spurious bridges. A post-processing baseline, described in the next section, completes the set.
Why not just…?
Why not just train the structure restorer with TopoMatch, since ground truth exists at training time? This is the real threat. If TopoMatch-trained models match our method on entry-pair connectivity at 60–70% mask ratios, then test-time refinement adds nothing. In that case the honest contribution shrinks to the diagnostic finding. The comparison is built in as the main baseline.
Our reason to expect a gap is distribution shift. Test masks and scenes differ from training ones, and a loss that can be minimized on the test image itself should help most where training-time supervision generalizes worst. We will show this by plotting the gap against mask ratio and against a mask-shape shift, training on irregular brush masks and testing on large block masks.
Why not just add clDice? It is cheaper than persistent homology. However, TopoMatch reports that it outperforms clDice on Betti error across six datasets. Lv et al. report that clDice still leaves the canal broken at its faded ends despite Dice 0.844. Both results predict that skeleton overlap alone will not hold connections across large holes. clDice is nonetheless run as a baseline rather than dismissed.
Why not just link dangling endpoints with a morphological post-process? Closing gaps between nearby endpoints with straight or minimal-cost paths is a strong, cheap competitor on roads, where paths are nearly straight. It knows nothing about which entries should pair, however, and it cannot change the texture-stage conditioning in a learned way. We run it on the naive sketch with tuned gap length and path cost. If it matches us on roads but not on membranes and indoor wireframes, which have more junctions, that defines the scope of the method.
Why not just enlarge the receptive field or re-weight attention, as iTaD does for language models? ZITS++ already uses Fourier convolutions and kernels up to 51×51, and it still fragments on large masks. That suggests reach is not the bottleneck. The distance-equalized attention variant tests this directly. If attention correction alone closes most of the gap, then the attention diagnosis is the main story and the loss is secondary.
Proposed Experiments
We use three settings in which the ground-truth sketch has meaningful topology:
- Indoor wireframes, the ZITS++ setting, with PSNR, FID and in-hole Betti error on the predicted sketch.
- Massachusetts Roads label maps, used in TopoMatch and TopoNet, with synthetic holes covering 30–70% of the image. Here the target is the road mask itself.
- CREMI membrane maps under the same masks.
For roads and membranes, the metrics are in-hole Betti-0 and Betti-1 error, entry-pair connectivity accuracy against ground truth, and clDice. For CREMI we add variation of information.
The first decisive check is the diagnosis. Compare the rank correlation between error-saddle locations and low anchor attention against the correlation with distance into the hole alone. The attention account must explain variance beyond distance.
The second decisive check is the stratified curve of Betti error against mask ratio for naive, clDice, TopoMatch, post-processing, and our method with and without test-time refinement. Our claim predicts that the gap over TopoMatch widens beyond 50% mask area and under mask-shape shift. If the curves coincide, the claim fails.
The third decisive check is a spurious-bridge audit. On held-out images where ground-truth lines truly terminate inside the hole, we count bridges falsely created by each method. This guards against the loss buying connectivity by inventing it.
References
- ZITS++: Image Inpainting by Improving the Incremental Transformer on Structural Priors
- Topologically Faithful Image Segmentation via Induced Matching of Persistence Barcodes
- Topology-Preserving Deep Image Segmentation
- TopoGAN: A Topology-Aware Generative Adversarial Network
- Mitigating Hallucinations in Multi-modal Large Language Models via Image Token Attention-Guided Decoding
- Vista-llama: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokens
- RAR: Reversing Visual Attention Re-Sinking for Unlocking Potential in Multimodal Large Language Models
- More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models
- Automatic segmentation of mandibular canal using transformer based neural networks