Scale-Matched Artifact Invariance for Few-Shot Degraded Document Segmentation

Many valuable images are photographs or scans of damaged paper, such as medieval manuscripts with ink seeping through from the reverse page, or thermal ECG printouts that are wrinkled, shadowed and faded. Hand-labeling these images is expensive, so practitioners train on synthetic pages: a program renders clean content and then applies simulated physical damage. Networks trained this way often score well on synthetic data and collapse on real paper. The literature also contains an unresolved tension. Networks with large receptive fields, meaning each output pixel can "see" a wide area of the input, help most when labels are scarce. Yet the same wide-view networks seem to be the ones that memorize the simulator's particular style of damage.

We propose to resolve this tension with one training signal that the simulator already gives for free. Every synthetic degraded page has an exactly matching clean page, and the simulator knows how large each applied artifact is. Our model has parallel branches with small and very large receptive fields. We train each branch to produce the same features on the clean and degraded versions of a page, and we weight this requirement most heavily for artifacts whose physical size matches what that branch can see. A large-view branch is pushed to ignore stains, folds and lighting gradients. A fine branch is pushed to ignore bleed-through speckle and print fading. The model keeps its wide context but is discouraged from storing the simulator's large-scale noise. We test this with 0 to 50 real labeled pages on historical document benchmarks and on real photographed ECG printouts.

Core Claim

The synthetic-to-real collapse in multi-scale networks comes from their large-context pathways encoding simulator-specific large-scale artifacts. Penalizing each pathway's sensitivity to artifacts at its own spatial scale, using clean/degraded pairs from a physics-style simulator, should close more of the gap than penalizing all pathways equally or not penalizing at all. It should also keep the receptive-field benefit that matters under few labels.

Two findings motivate this claim. ECGFusion trained a network with hierarchical dilated convolutions (filters with gaps between their weights, which widen the view without adding parameters) and spatial pyramid pooling on 50,000 images from ecg-image-kit. It reached an internal SNR of 7.273 but only −0.056 on the PhysioNet 2024 hidden test, and the authors attribute the drop to overfitting the synthetic artifact profiles. Diffusion Models and Masked Training… showed a similar drop, from 4.031 locally to −1.103 on the hidden test, after making the renderings more realistic with a diffusion model.

In the opposite direction, Dual-Kernel Adapter (DKA) found that under less than 1% training data, standard adapters, meaning small trainable modules added to a frozen pretrained model, shrink the model's effective receptive field. Its parallel 51×51 and 5×5 depthwise branches restore that receptive field and beat linear probing, reaching 89.01% versus 86.84% on COVID X-rays at 0.63% data. DKA was never tested on physically degraded inputs, and ECGFusion was never tested with few labels. Our proposal sits exactly where the two findings conflict.

Research Program

The backbone is a frozen pretrained vision transformer with a DKA adapter taken from the official code, misswayguy/DKA, plus a light segmentation head. We reuse DKA's two-branch design rather than inventing a new block. The contribution is the training objective, and keeping the architecture fixed isolates it. As a secondary architecture we test a cyclic-dilation layer in the style of PSConv, which mixes several dilation rates inside one layer at no parameter cost. PSConv was only ever evaluated on ImageNet and COCO, so this checks whether the effect depends on the branch design.

The simulators come from the reviewed literature. For documents we use a SynDoc-style generator in the manner of docExtractor, which places content on real blank historical scans and adds bleed-through, blur and yellowing. We extend it with the water stains and ink dilation described by Guan et al.. For ECGs we use ecg-image-kit, which renders PTB-XL signals with rotations, wrinkles, shadows and thermal fading.

We instrument each artifact so the simulator logs a characteristic scale $s_a$: for example stain radius, fold width, illumination-gradient wavelength or blur width. For every synthetic page the simulator emits the clean rendering $x$ and single-artifact variants $x_a$, where each variant has only artifact $a$ applied. The training objective is

$$ \mathcal{L} = \mathcal{L}_{\text{task}} + \lambda \sum_{b} \sum_{a} w(s_a, R_b)\, \lVert f_b(x) - f_b(x_a) \rVert_2^2 , $$

The symbols are defined as follows. $\mathcal{L}_{\text{task}}$ is the segmentation loss on real labeled pages together with fully degraded synthetic pages. $f_b$ is the feature map of branch $b$. $R_b$ is that branch's measured effective receptive field, taken from gradient maps rather than the nominal kernel size. $w$ is a weight that peaks when the artifact scale $s_a$ falls within the band $R_b$ covers. $\lambda$ sets the strength of the invariance term.

We also build a diagnostic, the per-branch artifact-sensitivity spectrum. It measures how much each branch's features move under artifacts of each scale, on both synthetic and real images. On real pages there is no clean pair, so we estimate sensitivity by feature variance across regions with and without annotated damage. The diagnostic lets us test the mechanism directly, instead of trusting the final metric alone.

Baselines

The naive reference point has two parts. The first is zero-shot transfer: a model trained only on synthetic pages, which is exactly what docExtractor reports. The second is linear probing of the frozen backbone on the N real pages, which DKA found hard to beat under scarce labels.

The strongest existing method on the historical benchmarks is docExtractor, trained purely on synthetic pages. The file reports an F-value of 0.900 on the cBAD2017 simple track and 76.1% mIoU on IlluHisDoc for it, in docExtractor. We rerun it with its published synthetic data and also fine-tune it on the same N real pages our method receives. Our method must beat both numbers at N = 0 and the fine-tuned version at every N.

For ECG digitization, the strongest reported system is the Hough-plus-nnU-Net pipeline of Krones et al., which won PhysioNet 2024 with an official SNR of 12.15. That number was obtained with about 21,799 synthetic training images on the challenge test distribution. Our main ECG comparison is therefore a rerun of that pipeline on our photographed-printout set, under the same synthetic budget and the same N real pages.

The most important baseline is DKA plus the same simulator with no invariance term. This is the "obvious combination." Every method gets identical synthetic volume, the same N real pages, three seeds, the same number of optimizer steps, and an equal hyperparameter search budget of 20 trials each, which covers DKA's separate learning rates for backbone, adapter and head.

We run two ablations to isolate the key ingredient. The first applies the same invariance loss with a uniform weight $w$, so every branch is made invariant to every artifact. The second applies invariance only on the final fused features. The proposal counts as successful only if the scale-matched version beats both ablations by more than the variation across seeds.

Why not just…?

Why not just use AugMix or generic noise instead of a physics simulator? Limitations of Synthetic Data Generation… found that AugMix matched or beat diffusion and GAN augmentation under data scarcity, so it is a serious competitor. However, that study covered global classification only. For dense segmentation of historical pages, docExtractor reports 11.5% mIoU for a model trained on modern layouts versus 76.1% with explicit bleed-through simulation. We include AugMix on DKA as a baseline. If AugMix matches the simulator, then the invariance term, not the simulator, must carry any gain, and we report that outcome.

Why not just strip artifacts with classical preprocessing? Hough-based rectification delivered the winning 12.15 SNR, but its authors report failures on phone photos and monitor captures, where perspective and lens distortion break the planar assumptions. Our photographed-ECG set targets exactly these conditions. We also report the combination of classical rectification followed by our model, because the two may be complementary rather than competing.

Why not just add a standard clean/degraded consistency loss? This is the real threat. Paired invariance without any scale structure is simple, and it may capture most of the benefit. The uniform-weight ablation tests exactly this. If it ties with our method, the scale-matching claim fails. What would remain is the diagnostic finding about which branch memorizes simulator noise, and we would report the contribution in those narrower terms.

Why not just linear-probe or fine-tune fully with the few real pages? DKA's results show that linear probing beats standard adapters but loses to receptive-field-preserving adapters at under 1% data. Whether this still holds on degraded inputs is unknown, so linear probing is our naive baseline at every N.

Proposed Experiments

The primary benchmarks are cBAD2017/2019 for text-line extraction and IlluHisDoc for illustration segmentation. We train with N ∈ {0, 5, 10, 25, 50} real labeled pages plus simulator data. The metrics are F-value and mIoU, and we additionally report the gap between synthetic validation and real test scores.

The secondary benchmark tests ECG digitization under distribution shift. We print about 200 held-out PTB-XL records, crumple and stain the printouts, and photograph them with phones and flatbed scanners. Because the underlying digital signals are known exactly, this yields true ground-truth signals. We score reconstructions with SNR, as in the PhysioNet challenge, and separately measure R-peak amplitude error. This second metric targets the peak smoothing that YOUR-Lead reported, since invariance losses risk erasing fine detail.

Three checks decide the question. First, the scale-matched loss must reduce the synthetic-to-real gap relative to DKA with no invariance term and relative to the uniform-weight ablation, while keeping at least DKA's effective receptive field. Second, the sensitivity spectrum should show that, without our loss, the large branch reacts strongly to large synthetic artifacts, and that across models this reaction predicts the size of the real-data gap. If no such correlation appears, the mechanism is wrong even if the final score improves. Third, a leave-one-artifact-out test removes stains, for example, from training and checks that invariance still transfers to real stains, which separates true invariance from memorizing the simulator.