Suppose you have a new historical manuscript and can afford to transcribe only a few dozen lines to train a handwriting recognizer. Which lines should you pick? The usual active-learning answer is to ask the model which lines it is unsure about. With no labels yet, that answer is meaningless, because the model's uncertainty is noise. Connected scripts make the problem worse. In cursive Latin, Arabic, or running-hand Chinese, the hard part of recognition is deciding where one character ends and the next begins. A recognizer trained on too few lines never learns those junctions, and a random handful of lines may contain few of the difficult ones.
We propose to score unlabeled lines before any annotation, using only the ink. We train a small image model to fill in narrow vertical strips of each text line from the surrounding context. Its per-column failure to fill a strip, the reconstruction residual, is high where the local stroke structure cannot be predicted from neighbouring ink. That tends to happen at ligatures, touching characters, and unusual joins. Two things follow from this map. First, we select the first annotation batch by residual mass balanced against coverage. Second, after that batch is transcribed, we aim an expensive handwriting generator only at the character pairs whose junctions carry high residual, rather than generating a uniform synthetic corpus.
Core Claim
A masked-strip reconstruction residual, computed without labels, is a better cold-start selection signal for connected handwriting than random sampling, whole-image pretext losses, or feature-space coverage. The claim is specifically about boundary difficulty. Lines chosen this way should contain more character junctions that a recognizer gets wrong, and a recognizer trained on them should reach lower character error rate (CER) at the same line budget. A secondary claim is that the same residual, aligned to characters, lets targeted synthetic generation match the gains of uniform generation at a fraction of the GPU hours.
The idea borrows its core logic from two places and changes both. PT4AL showed that a self-supervised pretext loss such as rotation prediction ranks unlabeled images by downstream difficulty, with Spearman ρ ≈ 0.8 on image classification. At cold start on CIFAR10 it reached 55.83% accuracy against 46.02% for random sampling. PT4AL produces one scalar per image for a discrete label, and it notes that the choice of pretext task is fragile across modalities. Rotation carries no information about where characters meet in a text line. Our pretext is spatially local and aligned with the reading direction, so its errors can be located along the line.
Reconstruction-based anomaly detection supplies the second ingredient. That line of work trains a generator on normal data and reads anomalies from where the reconstruction differs from the input. DRÆM localizes defects on industrial surfaces this way, pairing a reconstruction network with a discriminator trained on synthetic corruptions. The same literature warns of the "identical shortcut": an expressive model simply copies the anomaly, and the residual vanishes (Xing et al. respond by conditioning on normal templates). Masking the strip we score removes that shortcut by construction, since the model never sees the pixels it must predict.
Research Program
The scorer is a small convolutional or transformer inpainting network trained on the unlabeled pool itself. It receives a line image $x$ with a vertical strip of width $w$ centred at column $t$ blanked out, and it reconstructs that strip. Let $\hat{x}_t$ be the reconstruction and $m_t$ the ink mask of the strip. The column residual is the ink-normalized reconstruction error:
$$r_t(x) = \frac{\lVert m_t \odot (x - \hat{x}_t) \rVert_1}{\lVert m_t \rVert_1 + \epsilon},$$
Here $\odot$ is elementwise product and $\epsilon$ is a small constant for empty strips. The line score $s(x)$ is the mean of the top-$k$ values of $r_t$ over $t$. It rewards lines that contain several unpredictable junctions, and it does not reward lines that are uniformly long or dense.
Selection has to guard against the degenerate loop the file flags, where the scorer picks only outliers. To avoid it, we cluster lines in the scorer's encoder space and take the highest-scoring lines from each cluster in proportion to cluster size, as FeatProp does for coverage on graphs.
The main confound is physical degradation. Ink bleed, stains, and faded strokes also defeat inpainting, and Antonacopoulos et al. document how common they are in historical documents. We handle this by training a DRÆM-style discriminator on synthetic degradations only, meaning Perlin-noise blots and simulated bleed-through, and subtracting its predicted mass from $r_t$. What remains is meant to be structural residual. Whether that subtraction is needed, and whether it removes real boundary signal, is tested directly in the ablations below.
The second stage runs after the first batch is transcribed. A recognizer trained on the seed gives a CTC alignment for each line (connectionist temporal classification is the standard transcript-only training loss for sequence recognizers). That alignment maps residual peaks to adjacent character pairs, and we rank bigrams by accumulated residual. We then spend a fixed generation budget on words containing high-residual bigrams, using an existing few-shot writer-conditioned generator in the style of DiffusionPen or the autoregressive Emuru model used in Quo Vadis. The comparison is against the same budget spread uniformly. This stage is where generation overhead enters: GC-DDPM reports 62 hours on 8 GPUs to synthesize a character training set. On IAM, DiffPen's 55,000 synthetic words moved real-data CER only from 5.14% to 4.71%. The file also cites evidence from other domains that volume alone saturates (Beyond Quantity).
Baselines
The naive reference is random selection of the seed lines. We train the same DefCRNN recognizer used in Quo Vadis with identical augmentation, schedule, and tuning, and average over at least five random seeds, because random draws at 16 lines are high-variance. The strongest reported figure for this regime is from Quo Vadis. On Leopardi with 16 real lines (1.25% of the data), CER is about 62% trained from scratch and 21.7% with Emuru synthetic pretraining. The proposal beats this pipeline if, with the same Emuru pretraining and 16 lines, residual-selected lines give CER clearly below 21.7%, outside the random-seed spread, or match 21.7% with substantially fewer lines. On Washington and Saint Gall, Quo Vadis reports zero-shot CER of 15.3% and 18.1% with Emuru. Any selected batch must improve on those zero-label floors to be worth annotating at all.
The strongest competing selection methods get the same encoder capacity and pool. The first is PT4AL run as published, using rotation-prediction loss on line images. The second is coverage-only selection: k-means centres in the embedding of a self-supervised handwriting encoder in the spirit of LoGo-HTR, trained at our pool scale rather than its 10-million-image scale, so that pretraining compute stays matched. The third is confidence ranking from an off-the-shelf pretrained recognizer, using the TrOCR readability score that Quo Vadis uses as a synthetic-data filter. For these on handwritten lines, we found no reported result on Lacuna for cold-start active learning.
Three ablations isolate the key ingredient. The first replaces masking with plain autoencoding, which tests the identical-shortcut argument. The second scores by mean residual instead of top-$k$, which tests the junction-concentration assumption. The third removes the degradation subtraction.
For the generation stage, the baseline is uniform generation plus TrOCR filtering as in Quo Vadis, at equal generator GPU hours, including filtering time. Winning means equal or lower CER at half the hours or less.
Why not just…?
Why not just sample the seed randomly and rely on synthetic pretraining? This is the most serious threat. Quo Vadis shows that Emuru pretraining alone takes Leopardi from about 62% to 21.7% CER with random lines, and a strong prior may leave little for selection to add. We test selection on top of that pretrained model. If the gap to the random-seed spread is within noise, the selection claim fails for pretrained pipelines, and we will report it that way.
Why not just use PT4AL's rotation loss? PT4AL itself notes that pretext proxies are sensitive to modality. Rotation difficulty on a text line mostly reflects aspect ratio and line length, not junctions. The direct test is the per-junction check below. If rotation loss tracks error-prone junctions as well as masked-strip residual does, our localization adds nothing.
Why not just ask a pretrained recognizer which lines it is unsure of? On scripts close to its training data this may work well. On historical hands, the file's sources describe early uncertainty as uncalibrated and prone to optimism bias (Introduction to Active Learning; Liu et al.). We expect the TrOCR baseline to be competitive on IAM and weaker on Saint Gall and Leopardi. That dataset-dependent pattern is itself a finding, and it tells practitioners when to prefer the ink-only score.
Why not just select for diversity in a self-supervised embedding? Coverage alone avoids outliers but ignores difficulty, and selection criteria need to balance the two. The coverage-only baseline and our cluster-stratified variant differ only in within-cluster ranking. That makes the comparison a clean test of whether the residual adds information beyond geometry.
Proposed Experiments
The main experiment covers Leopardi, Washington, Saint Gall, and IAM. We use seed budgets of 16, 32, and 64 lines, with and without Emuru pretraining, and report CER and word error rate. For context on IAM, LoGo-HTR reports 2.31% CER with 80% of the labels.
To test the boundary mechanism directly, we use Chinese text-line data from CASIA-HWDB and ICDAR2013-Offline, where character positions are available and touching characters are common. There we measure whether residual peaks fall on true inter-character junctions (precision and recall of peaks within a tolerance of character boundaries). We also measure whether selected lines contain more junctions that a random-seed recognizer later misreads. SegCTC (SegCTC) reports AR 95.10% without a language model on ICDAR2013. We use it only as a full-supervision ceiling, not as a competitor.
The decisive check is as follows. If residual-selected seeds do not beat random, rotation, coverage, and TrOCR confidence, or if their residual peaks do not concentrate on junctions any more than degradation-only scores do, the core claim is false. The generation stage would then be reported as unmotivated rather than tuned until it wins.
For the generation stage, we plot CER against generator GPU hours on IAM and Leopardi, comparing bigram-targeted generation with uniform generation and filtering. The measure of interest is the area under that curve, which captures whether residual targeting reduces the generation overhead that GC-DDPM and DiffPen make explicit.
References
- PT4AL: Using Self-supervised Pretext Tasks for Active Learning
- DRÆM - A discriminatively trained reconstruction embedding for surface anomaly detection
- A recover-then-discriminate framework for robust anomaly detection
- Active Learning for Graph Neural Networks via Node Feature Propagation.
- A Complete Approach to the Conversion of Typewritten Historical Documents for Digital Archives
- DiffusionPen: Towards Controlling the Style of Handwritten Text Generation
- Quo Vadis Handwritten Text Generation for Handwritten Text Recognition?
- Zero-shot Generation of Training Data with Denoising Diffusion Probabilistic Model for Handwritten Chinese Character Recognition
- Beyond Quantity: Trajectory Diversity Scaling for Code Agents
- Learning Beyond Labels: Self-Supervised Handwritten Text Recognition
- Introduction to Active Learning and Application to Material Engineering
- Shift-Pessimistic Active Learning Using Robust Bias-Aware Prediction
- SegCTC: Offline Handwritten Chinese Text Recognition via Better Fusion Between Explicit and Implicit Segmentation