When text ink is only slightly darker or lighter than what lies behind it, as with faded signage, glow effects, or brush strokes on textured paper, pixel-level text segmentation fails in two characteristic ways. Strokes break into pieces, and the small background regions enclosed by letters such as "O", "B" and "A" get filled in. This proposal argues that both failures come from asking the network the wrong question. A standard segmentation network labels every pixel "ink" or "not ink." At low contrast, that forces it to guess where each edge lies, and on a stroke three pixels wide both edges are guessed at once. The network could instead answer two easier questions. The first is where the stroke's centerline runs. The centerline is a ridge of the local contrast and stays detectable even when the edges blur. The second is how wide the stroke is at that point. Width changes slowly along a stroke, so a network can carry it from clearly visible segments into faint ones. The mask is then drawn from centerline and width rather than predicted pixel by pixel.
Concretely, we build a U-Net, an encoder-decoder network whose skip connections pass high-resolution features from the downsampling path to the upsampling path. Its decoder outputs a centerline (skeleton) probability map and a local half-width map. A small differentiable renderer turns these into a mask by taking the union of discs placed along the centerline. All training targets come from existing ground-truth masks through the medial-axis transform, so no character boxes, recognizers, or new annotation are needed. The claim is specific and falsifiable. On the lowest-contrast strokes of TextSeg and BTS, this decoding should lose fewer holes, break fewer strokes and place boundaries better than a U-Net with the same backbone, loss budget and auxiliary skeleton supervision that predicts the mask directly. It should also match that U-Net on high-contrast text.
Why this problem, and why not the obvious combination
Three facts from prior work motivate the proposal. Rethinking Text Segmentation (TexRNet) showed that generic segmenters blur thin, non-convex glyph boundaries. It attacks this with a boundary-band "trimap" loss and a frozen character classifier that criticizes non-letter shapes, which requires character boxes during training. BTS (PGTSNet) adds a text detector and a large text recognizer as perceptual critics. Char-SAM documents that the Segment Anything Model fills character holes unless an 80-font template library supplies negative point prompts. All three add external knowledge about letters. None of them changes what the segmenter outputs.
The obvious way to combine the three directions is to move reverse attention from polyp segmentation into a U-Net for text. Reverse attention, introduced in PraNet and extended in MISNet, erases high-confidence object interiors so that later decoder stages attend only to the uncertain rim. That logic assumes blob-shaped objects whose interior is much larger than their boundary band. A text stroke of width two to five pixels has no confident interior to erase, because the boundary band is the whole stroke. We therefore treat reverse attention as a baseline to beat, not as the mechanism.
The proposal also builds on an older idea. A Stroke-Based Text Extraction Algorithm for Digital Videos estimated stroke width from pairs of parallel edges using histograms. That requires both edges to be visible, which is exactly what fails at low contrast. We keep the width variable but learn it, and we let it propagate along the centerline instead of measuring it from edges.
Evidence from thin curvilinear structures outside text supports the diagnosis. The five-U-Net ensemble in the corneal nerve study of Williams et al. still broke single nerve fibers into fragments under uneven illumination, and the authors called for external gap filling. This is the fragmentation failure we target. Because the failure is shared, nerve fibers also serve as a second test domain.
Core Claim
Predicting a stroke as a centerline plus a width is better conditioned than predicting its pixels whenever edge contrast is low. The centerline tolerates edge blur. The width can be learned from high-contrast parts of the same stroke. Drawing the mask from these two quantities keeps holes open by construction whenever the predicted width is smaller than the hole's radius. We expect the gains to be concentrated in the low-contrast bins and to disappear as contrast rises. That concentration is the signature that separates a real mechanism from a generic capacity gain.
Research Program
Network and outputs. The network is a U-Net with a ResNet-style encoder. We also port the same head onto HRNetV2-W48 so it can be compared directly with TexRNet. At full resolution the decoder emits a skeleton probability $s(p)$ and a half-width $r(p) > 0$ for every pixel $p$. The rendered soft mask is
$$ m(x) = \max_{p \in N(x)} \; s(p)\,\sigma\!\left(\frac{r(p) - \lVert x - p \rVert}{\tau}\right), $$
where $x$ is a pixel, $N(x)$ is a local window of radius $R$ around it (set to the 99th percentile of ground-truth half-widths), $\sigma$ is the logistic function, and $\tau$ is a softness temperature annealed toward one pixel. The max is computed with a sliding-window operation, so its cost is comparable to one extra convolution layer.
Training targets. For each ground-truth mask we compute the medial-axis skeleton and the distance-transform value along it, which gives target half-widths $r^*$. The loss has four terms. The first is a class-balanced cross-entropy on $s$ against the skeleton, dilated by one pixel. The second is an L1 loss on $r$ at skeleton pixels. The third is Dice on the rendered mask $m$ against the full mask; this is where the renderer receives gradient. The fourth term makes width travel along strokes. It is a smoothness penalty along the local stroke direction:
$$ L_{\text{smooth}} = \sum_{p} s(p)\,\big|\, t(p)^\top \nabla r(p) \,\big|, $$
where $t(p)$ is the unit tangent of the skeleton at $p$, estimated from the structure tensor of $s$. The penalty discourages width from jumping along a stroke while allowing it to change across strokes and to taper gradually, as serifs and brush strokes do. We ablate this term separately.
What the U-Net contributes. The skeleton of a two-pixel stroke exists only at full resolution, so the high-resolution skip connections are what keep it alive. This is the reason the design uses a U-Net rather than PraNet's partial decoder, which drops low-level encoder features. The design resolves the tension the research file identifies between pruning low-level features to suppress noise and keeping them for thin structures. Low-level features are kept, but they feed a sparse, one-pixel-wide target. A background texture then has to produce a coherent ridge together with a plausible width before it contributes ink.
Low-contrast stratification. Contrast stratification is how the claim is measured, so it is defined up front. For each ground-truth stroke pixel we compute local contrast as the CIELAB color difference between the mean over a two-pixel band inside the stroke and the mean over a ring just outside it. Test strokes, taken as connected components within character regions, are binned into contrast quintiles. Per-bin metrics are the central evidence.
Topology metrics. Hole preservation is the fraction of ground-truth enclosed background components that remain enclosed background components in the thresholded prediction. Fragmentation is the absolute error in the number of connected ink components per word region. Boundary quality is the boundary F-score at a one-pixel and a two-pixel tolerance. We report the standard foreground IoU (fgIoU) and F-score alongside, for comparability.
Baselines
The naive reference point is a plain U-Net with the same encoder, trained with soft Dice plus class-weighted cross-entropy on the mask. This is the stroke-mask branch of SAEN, which used exactly this setup and still needed a second refinement stage to remove residual text. Any proposal that cannot beat it in the lowest contrast quintile has failed.
The ablation that isolates the key ingredient is the most important comparison. It is the same U-Net, with the same skeleton and width heads and the same losses on them, but its mask comes from a direct per-pixel head rather than from the renderer. This separates "rendering the mask from geometry" from "multitask learning with skeleton supervision." The research file does not report this configuration, so it has no published number. Beating it means the renderer version shows higher hole preservation and lower fragmentation in the bottom two contrast quintiles, with non-overlapping confidence intervals across three seeds. A second ablation removes the along-stroke smoothness term to test whether width propagation, rather than the skeleton alone, drives the gain.
The strongest boundary-focused competitors come in two forms. The first is TexRNet's trimap loss and key-feature attention on a U-Net. The second is a reverse-attention U-Net built with PraNet-style foreground erasure inside each skip connection. Both are reimplemented on our encoder with the same data, number of epochs, augmentations and tuning budget, meaning an equal number of learning-rate and loss-weight trials for every method. On TextSeg, the published target is TexRNet with HRNetV2-W48 at 86.84% fgIoU, reported in Rethinking Text Segmentation. Char-SAM cites supervised TFT at 87.11% fgIoU, and zero-shot Char-SAM reports 84.80% fgIoU. On BTS, the published target is PGTSNet at 86.48% fgIoU and 90.98% F-score, from BTS. We do not claim beforehand to exceed these overall numbers with a lighter U-Net. For a fair headline comparison we attach our head to HRNetV2-W48. Beating TexRNet means matching it within half a point of overall fgIoU while clearly improving the low-contrast quintiles and the topology metrics, without character boxes or a recognizer. For text removal, the reference is SAEN at 34.75 PSNR on SCUT-EnsText, from Modeling Stroke Mask for End-to-End Text Erasing. We substitute our mask into SAEN's two-stage pipeline and leave everything else fixed.
Why not just…?
Why not just add a boundary or trimap loss to a plain U-Net? This is a real threat. TexRNet's trimap loss already focuses supervision on the ink-to-background transition, and Niu et al. show that contour losses regularize medical segmenters. The argument against it is that a boundary loss still asks for edge locations the image barely supports, and it contains nothing that transfers width from a clear segment to a faint one. We do not assume this argument holds. The trimap U-Net is run under the same budget. If it matches the renderer in the bottom contrast quintiles, the proposal's central claim is wrong and we will report that.
Why not just use reverse attention inside the skip connections? That is the obvious combination of the three directions. Its mechanism erases a confident interior that thin strokes do not have. PraNet was validated only on blob-shaped polyps, and it fell to 0.628 Dice on the unseen ETIS set. The direct test is the per-width breakdown. Reverse attention should do comparably on thick display text and fall behind on strokes narrower than about four pixels. If it does not fall behind, reverse attention is the simpler answer.
Why not just supervise a skeleton as an auxiliary task and keep the direct mask head? Multitask skeleton supervision might transfer most of the benefit into the shared features, which would make the renderer unnecessary. This is exactly the key ablation described above. It is the experiment most likely to deflate the proposal, which is why it gets three seeds and per-quintile confidence intervals.
Why not just iterate the network or prompt a foundation model? RecycleNet refines ambiguous boundaries by feeding a U-Net's outputs back into its earlier layers. Its cost grows with each pass, and its refinement is still pixel-wise. We test it as an orthogonal addition: recycling applied to our model versus recycling applied to the direct-mask U-Net. Char-SAM needs a character detector and a font library to keep holes open, and it degrades on calligraphy and non-Latin scripts. Our hole-preservation metric on the Chinese portion of BTS directly tests whether rendering from width recovers holes without any font prior.
Proposed Experiments and Decisive Checks
The primary experiment trains every model on TextSeg and on BTS separately, using the standard splits. We report overall fgIoU and F-score, then per-contrast-quintile hole preservation, fragmentation and boundary F-score. The decisive check is the interaction between method and contrast. The renderer's advantage over the direct-mask ablation should grow monotonically as contrast falls and be near zero in the top quintile. A uniform gain would point to extra capacity, not the proposed mechanism.
The second experiment checks whether the renderer confuses scale with contrast. We synthetically reduce contrast in held-out TextSeg crops by blending the ink toward the local background color in graded steps. Degrading the same strokes step by step isolates contrast from font and scale. We plot fgIoU and hole preservation against blend factor for every baseline.
The third experiment tests whether the mechanism is specific to text or general to thin structures. We train on the corneal nerve images used by Williams et al., from the BioImLab datasets. We measure fragment count per fiber and nerve-length agreement against their reported total-nerve-length intraclass correlation of 0.933 for the LDLA ensemble. We use a single model rather than their five-model ensemble. The question is whether width propagation alone closes the gaps they had to fill by post-processing.
The fourth experiment measures downstream value: SCUT-EnsText PSNR when our mask replaces SAEN's stroke mask. A thinner, hole-preserving mask should leave less for the inpainting stage to invent.
The failure analysis inspects where the method breaks. We expect breakdowns where a stroke's width changes abruptly, such as ink blots and decorative flourishes, and where two strokes touch so that their skeletons merge. Both cases are counted separately so that the claim's scope is stated honestly.
References
- Rethinking Text Segmentation: A Novel Dataset and a Text-Specific Refinement Approach
- BTS: A Bi-lingual Benchmark for Text Segmentation in the Wild
- Char-SAM: Turning Segment Anything Model into Scene Text Segmentation Annotator with Character-level Visual Prompts
- PraNet: Parallel Reverse Attention Network for Polyp Segmentation
- Multi-scale information sharing and selection network with boundary attention for polyp segmentation
- A Stroke-Based Text Extraction Algorithm for Digital Videos
- An artificial intelligence-based deep learning algorithm for the diagnosis of diabetic neuropathy using corneal confocal microscopy: a development and validation study
- Modeling Stroke Mask for End-to-End Text Erasing
- A Medical Image Segmentation Method based on Multi-scale Features and Contour Loss Constrain (S)
- Abstract: RecycleNet - Latent Feature Recycling Leads to Iterative Decision Refinement