Cross-Occurrence Noise2Noise: Training Document Denoisers on a Page's Own Repeated Letters

A printed or typewritten page contains hundreds of copies of the letter "e", and all of them came from the same piece of metal type or the same typewriter hammer. Each copy is the same shape seen through different damage: a stain here, faded ink there, ghost text from the reverse side somewhere else. We propose to use pairs of these copies as training data for a neural image denoiser. The network gets one damaged occurrence as input and a different damaged occurrence of the same letter as its target, so it is trained without any clean reference image. The underlying idea is Noise2Noise, a known result that a network trained to map one noisy picture of a scene to another independent noisy picture of the same scene learns to output the clean scene. The part that is new is where the second picture comes from. Pairing across different locations on the page makes damage that is tied to a location, such as bleed-through, stains and foxing, statistically unrelated between input and target. Location-tied damage of this kind is exactly what defeats existing single-image self-supervised denoisers.

This matters because the problem is not image quality for its own sake. Character errors from OCR (optical character recognition, the conversion of page images to text) break subword tokenization and corrupt downstream language models. ChronoLens documents how transcription noise contaminates embedding-based measurements of language change. Yet every strong deep document restorer we found (DocEnTr, DocBinFormer, DE-GAN, DocPure) needs paired clean targets, and such targets do not exist for real historical archives. The DocEnTr authors name self-supervised learning on unlabeled historical data as their next step. We propose a specific way to take it, and we build it so that it survives the obvious objection that one could simply average the repeated glyphs.

Core Claim

Single-image self-supervised denoisers such as Noise2Void and Neighbor2Neighbor assume the noise at a pixel is independent of the noise at neighboring pixels. Noise2Void does this by hiding each pixel and predicting it from its surroundings, a so-called blind-spot network. Neighbor2Neighbor does it by pairing interleaved sub-images of one image. Bleed-through breaks this assumption, because ghost text is spatially coherent and looks like real strokes. Both methods will therefore reproduce it as signal. Blind-spot methods have a second problem with text: they struggle with details narrower than the masked region. Strokes are one to three pixels wide, so masking the center of a stroke leaves mostly blank paper as context. Noise2Void itself reports that fine structure is lost.

Our claim is that pairing across occurrences fixes both problems. Take two aligned occurrences $y_i$ and $y_j$ of the same glyph at page locations $i$ and $j$. The ghost text behind $y_i$ comes from whatever is printed on the verso at $i$, which has no relation to what is printed at $j$. Each input therefore carries structured damage, but that damage is independent of the damage in its target. The network sees the full input without masking, so thin strokes remain observable.

The training objective is $$\mathcal{L}(\theta) = \sum_{(i,j)\in\mathcal{P}} \big\| M_{ij} \odot \big(f_\theta(c_i) - \mathcal{A}_{ij}(c_j)\big) \big\|_1 .$$ The terms are defined as follows. $\mathcal{P}$ is the set of occurrence pairs judged to show the same glyph. $c_i$ and $c_j$ are image windows centered on the two occurrences. $\mathcal{A}_{ij}$ is a sub-pixel alignment that registers the target onto the input. $M_{ij}$ is a mask restricting the loss to the region where the two windows actually share content. $f_\theta$ is a fully convolutional denoiser with parameters $\theta$. We use $L_1$ rather than $L_2$ because $L_1$ converges to a median, not a mean, which gives some robustness to mispaired targets. Noise2Noise showed that $L_1$ tolerates outlier-heavy targets.

The estimator is biased in one known way. Averaged over random page locations, bleed-through and paper tone are not zero-mean, so the network converges to the glyph plus a roughly uniform background tint. That tint is removed by global intensity normalization. The Noise2Noise zero-mean failure noted in the file becomes an offset we can correct, rather than ghost strokes left in the output.

Relation to Prior Work

Two lines of work already exploit the fact that glyphs repeat. They motivate this proposal and are also its strongest competitors.

Improving OCR using internal document redundancy crops character boxes from a base OCR engine and normalizes their backgrounds with Poisson editing. It clusters the crops with PCA and a Gaussian mixture, splits clusters that fail an Anderson–Darling normality test, and relabels each cluster by super-majority vote. It improves labels but never restores pixels. It only corrects substitutions, and it cannot act where the base OCR failed to find a character box.

Unsupervised Transcription of Historical Documents (Ocular) learns per-document glyph templates within a generative model of typesetting, inking and binary pixel noise, trained by expectation-maximization. It explicitly leaves grayscale pixels and complex backgrounds open.

Both methods reduce redundancy to a per-glyph template or a label decision, which means committing to a character identity. Our method instead learns a general image-to-image map. After training, that map applies to every pixel of the page: undetected glyphs, merged characters, and punctuation. It produces a restored image that any OCR engine can read. It also never commits to an identity. Where the input is truly ambiguous between "b" and "h", it outputs an honest blend rather than the confidently wrong template.

Font Representation Learning via Paired-glyph Matching supplies an ingredient. It trains contrastive embeddings in which glyphs of the same font sit close together, though only on clean digital fonts. We use a similar embedding for pair mining on degraded scans. A Non-parametric Framework for Document Bleed-through Removal removes bleed-through with a Markov random field over registered recto–verso scans. It constrains our design: it shows that bleed-through is separable when the second side is available. We target single-sided archives where it is not, and we use its benchmark as a check on whether our method removes ghost text.

Research Program

The system runs in three stages. First, a base OCR engine proposes character boxes on each page. We define pairs from visual similarity (aligned crops close in a paired-glyph embedding and in pixel correlation) rather than from the OCR labels alone, so that OCR mistakes do not become pairing mistakes. Frequent whole words such as "the" and "and" give larger shared regions and are paired the same way.

Second, a U-Net denoiser is trained per collection. We also test per-page training to measure how much redundancy is enough. Training uses the loss above, with $M_{ij}$ set to the aligned glyph box expanded to include inter-letter background whenever neighboring characters also match.

Third, the trained denoiser is applied to whole pages, and OCR is run again. The better transcription yields more and cleaner boxes, and we retrain on the resulting pair set. We measure whether this bootstrapping converges or drifts.

The analysis has one central empirical question. We test whether cross-occurrence pairs actually decorrelate the damage. We estimate the correlation between input and target residuals, computed against a DIBCO ground-truth mask, for cross-occurrence pairs and for Neighbor2Neighbor-style sub-sampled pairs. We break this down by degradation type. If cross-occurrence residuals are not markedly less correlated in bleed-through regions, the core claim fails regardless of downstream numbers. A second question is scope. Handwriting has no shared physical source, so we expect the method to fail on handwritten DIBCO images. We report that failure rather than hide it.

Baselines

The naive reference point is running Tesseract and a second engine on the raw scans. On Old Bailey, Unsupervised Transcription of Historical Documents reports Tesseract at 54.8% WER and ABBYY FineReader at 40.0%. On Trove the figures are 59.3% and 49.2%. We rerun Tesseract ourselves on identical page splits so every method is compared under the same engine version.

The strongest label-level competitor on this exact question is internal-redundancy relabeling from Improving OCR using internal document redundancy. It reports a CER reduction of "more than 1.0 percentage point" for CharNet on both Berrutti and ENP, and 31–34% correction accuracy on Google Cloud Vision output. We reimplement it on the same base OCR boxes we use for pairing. We count it as beaten if our restored images lower CER by more than it does on Berrutti and ENP. We also test whether the two methods stack.

The strongest transcription competitor is Ocular. It reports 24.1% WER on Old Bailey and 33.0% on Trove, and beating those figures with an off-the-shelf engine on our restored images would be a strong result. We do not require it, because Ocular also uses a language model.

The strongest self-supervised image baseline is Noise2Void, with Neighbor2Neighbor alongside it. Both are trained on the same pages with the same U-Net, the same number of gradient steps, and the same hyperparameter search budget. No reported result was found on Lacuna for Noise2Void or Neighbor2Neighbor on document images; the only reported Noise2Void figures are 27.71 dB PSNR on BSD68 and 32.28 dB on simulated microscopy.

The supervised ceiling is DocEnTr, which reports PSNR 19.11 and F-measure 92.53 on DIBCO 2017. DE-GAN reports 24.9 PSNR on DIBCO 2013 and a Tesseract CER drop from 37% to 1% on its degraded set. We run these models under two conditions: trained in-domain on DIBCO, and transferred with synthetic training to Old Bailey, Trove and ENP, where no clean pairs exist.

The key ablations isolate what the pairs contribute. The first replaces cross-occurrence targets with random other glyphs, which should fail. The second uses pairs formed only from OCR labels rather than visual mining. The third replaces the learned map with direct template substitution, meaning the cluster median is pasted in place of each glyph.

Why not just…?

Why not just paste each glyph's cluster average back into the page? This is the most serious threat. On clean, well-segmented pages, template substitution may match our method, because a Noise2Noise network on one glyph class converges toward that class's conditional average anyway. Where substitution should lose is on characters the base OCR mislabeled or never detected. Substitution turns a mislabel into a crisp wrong letter, and it cannot touch undetected characters. Belzarena et al. themselves note that their method depends on detected boxes and handles only substitutions. The template ablation, broken down by whether the base OCR had detected and correctly labeled each character, settles this question. If substitution matches us on all subsets, the learned network adds nothing, and we will say so.

Why not just use Noise2Void or Neighbor2Neighbor on each page? Both treat spatially coherent damage as predictable signal, and bleed-through is spatially coherent. Blind-spot masking also hides the thin strokes that carry character identity. The residual-correlation analysis and the bleed-through breakdown on the recto–verso benchmark of Rowley-Brooke et al. test this directly. We predict comparable results on pure sensor grain and a clear gap wherever ghost text is present.

Why not just train a supervised restorer on synthetic degradations? Such restorers do well in-domain, but the literature in the file stresses that synthetic noise models miss real sensor and material statistics (Learning Physics-Informed Noise Models from Dark Frames). DocPure also notes that real paired degradations cannot be collected. Transferring DocEnTr and DE-GAN to Trove and ENP measures this gap. If they transfer well, our method's value shrinks to settings without any clean fonts at all.

Why not just correct the OCR text with a language model? Text correction cannot recover proper nouns or archaic forms whose characters were destroyed. The glyph shape is evidence that exists only in the image. Text correction is also complementary to our method rather than a replacement. We report it applied on top of both raw and restored OCR output, so any gain from the images can be read directly.

Decisive Checks

We run everything on Old Bailey and Trove (WER), on Berrutti and ENP (CER), and on the printed subsets of DIBCO and H-DIBCO (PSNR and F-measure after thresholding). We use two OCR engines to guard against gains that hold for one engine only.

Three outcomes would falsify the proposal. The first is residual correlation for cross-occurrence pairs that is no lower than for sub-sampled pairs in bleed-through regions. The second is template substitution matching our CER on characters the base OCR had missed or mislabeled. The third is no CER gain over redundancy relabeling on Berrutti and ENP.

A positive result would be the first deep document restorer trained without clean data that beats both blind-spot self-supervision and label-level redundancy methods on real archives, with the gains concentrated where the mechanism predicts: in regions with structured, location-bound damage.