Legibility Priors: Recognition-Trained Denoisers for Plug-and-Play Document Restoration

Scanned documents arrive damaged in ways that keep changing: a different scanner's blur, a lower capture resolution, ink from the back of the page showing through. Today's best document restorers are single neural networks trained end to end on a fixed menu of damage types, so a new kind of damage means retraining. Plug-and-play (PnP) restoration avoids that. It is an iterative method that alternates between two steps: one enforces consistency with an explicit physical model of the damage, and the other runs a pretrained image denoiser that acts as a learned prior over clean images. Swapping in a new damage model requires no retraining. The weak link is the denoiser. Standard PnP denoisers are trained on photographs and learn that thin, high-contrast strokes look like noise. Even when they are retrained on documents, they are trained to minimize pixel error, and pixel error is known to be a poor proxy for whether an optical character recognition (OCR) engine can read the result.

We propose to train the denoiser inside the PnP loop for legibility rather than pixel fidelity. Training adds a loss from a frozen text recognizer to the usual reconstruction loss. Because the PnP structure keeps the denoiser separate from the physics, this recognition-aware prior can be reused unchanged across blur, downsampling and bleed-through. The question we test is concrete: under damage types that a trained end-to-end restorer never saw, does recognition-aware PnP produce images that independent OCR engines read with fewer errors, and do those gains survive into downstream language tasks such as named entity recognition?

Core Claim

Two properties together should make recognition-aware PnP win on character and word error rate when the damage operator shifts away from what an end-to-end restorer was trained on. The first is modularity: the physical damage model is explicit and replaceable. The second is a prior that encodes which pixel changes matter to a reader. Each property alone should fall short. A recognition-aware feed-forward restorer is still tied to its training damage. A pixel-trained document denoiser in PnP still smooths strokes in ways that OCR punishes. The proposal stands or falls on the interaction between the two, and the ablations are designed to isolate it.

The relevant prior work falls into three groups.

The first group supplies the method. Plug-and-Play Image Restoration With Deep Denoiser Prior introduced DRUNet, a bias-free denoiser conditioned on a noise-level map and used inside half-quadratic splitting. We reuse its architecture and its iteration schedule, but we replace natural-image training with document training plus a recognition loss. Provably Convergent Plug & Play Linearized ADMM, applied to Deblurring Spatially Varying Kernels handles spatially varying blur using only forward and adjoint blur applications. We adopt it as the data-consistency solver for our hardest operators.

The second group motivates the work. Building OCR/NER Test Collections shows that a 2.3% character error rate still drops NER F1 from 68.51 to 64.47, so small recognition gains matter downstream. DocWaveDiff and DocPure both report OCR alongside PSNR, which shows the field already distrusts pixel metrics alone. DocPure also illustrates the rigidity problem: it covers four fixed degradations.

The third group constrains the design. Plug-and-Play Methods Provably Converge with Properly Trained Denoisers guarantees convergence only for denoisers whose residual is contractive, and that condition conflicts with the near-binary sharpening that documents want. We therefore treat convergence empirically, using a fixed iteration budget and DRUNet's annealed noise schedule. We report a spectrally normalized variant only as a secondary check, and we do not claim guarantees.

The closest existing bridges are narrower. Blind image separation for document restoration using plug-and-play approach runs PnP for recto/verso bleed-through separation, but it uses only classical denoisers (BM3D, non-local means, TV), synthetic noise-free mixtures and no OCR evaluation. Uni-DocRobust is "plug-and-play" only in the sense of a module inserted into multimodal-LLM features. It produces no image, so it cannot serve standalone OCR engines, and its authors report that it struggles with geometric and physical damage. Deblurring Text Images via L0-Regularized Intensity and Gradient Prior runs the same splitting loop with a hand-designed two-tone prior and evaluates only image metrics.

Research Program

The denoiser. We build a DRUNet-style denoiser $D_\sigma$, meaning a network that takes a noisy image and a noise level $\sigma$ and returns a cleaned image. It is trained on text-line crops $x$ that have ground-truth transcripts $t$. The training objective is

$$ \mathcal{L}(D) = \mathbb{E}_{x,\sigma,n}\Big[\|D_\sigma(x+\sigma n) - x\|_1 + \lambda\, \mathrm{CTC}\big(R(D_\sigma(x+\sigma n)),\, t\big)\Big], $$

where the terms are defined as follows. $n$ is standard Gaussian noise. $R$ is a frozen line recognizer trained with connectionist temporal classification (CTC), the standard alignment-free sequence loss for text lines. $\lambda$ trades pixel fidelity against legibility.

The training deliberately follows the plain Gaussian-denoising recipe. Its purpose is to keep the network a valid plug-in for PnP at any noise level $\sigma$, independent of which damage it will later face. The only document-specific addition is the recognizer term. The weight $\lambda$ is ramped down as $\sigma \to 0$, so late iterations favor fidelity and the network cannot invent characters.

The physics. We implement four forward models in one half-quadratic-splitting codebase. The first is uniform blur with a closed-form fidelity step. The second is spatially varying blur via linearized ADMM. The third is two- to four-times downsampling with blur. The fourth is a noisy recto/verso linear mix that extends the separation formulation of Blind image separation for document restoration using plug-and-play approach by replacing its classical denoisers with ours. For blind settings, we alternate kernel estimation with image updates, as Deblurring Text Images via L0-Regularized Intensity and Gradient Prior does.

The evaluation protocol. The key design choice is that the recognizer used in training never scores the outputs. All error rates are measured with Tesseract and with at least one held-out recognizer of a different architecture. This guards against the denoiser learning patterns that fool $R$ specifically, which is a real risk with any loss computed through a frozen network.

Data. We use DIBCO and H-DIBCO ground truth and TDD clean patches for training and in-distribution tests. For operator-shift tests, we apply our four synthetic operators, and combinations of them, to held-out clean pages that have transcripts. For real historical print we use Old Bailey and Trove. For the downstream language test, we degrade the Renmin pages from Building OCR/NER Test Collections synthetically and score NER F1. The operator families are split so that each end-to-end baseline is trained on some families and tested on others.

Baselines

The naive reference. We run Tesseract and the held-out recognizer directly on the degraded image, with no restoration. This anchors every gain. On real historical print, Unsupervised Transcription of Historical Documents reports Tesseract WER of 54.8% on Old Bailey and 59.3% on Trove, ABBYY at 40.0% and 49.2%, and their generative model Ocular at 24.1% and 33.0%. We report against these figures, while noting that Ocular adapts to fonts without supervision, which our method does not.

The strongest end-to-end restorer. This is DocPure (DocPure), which reports 32.55 dB PSNR and 61.27% OCR word accuracy on TDD deblurring. DocWaveDiff reports Tesseract CER of 0.1316 and WER of 0.6495 on TDD. On binarization, DocEnTr reports F-measure 92.53 on DIBCO 2017. These are the strongest competitors because they are document-specific and are already evaluated on OCR.

For fairness, each end-to-end baseline is retrained on exactly our training pages, with the same degradation families our training sees and the same GPU-hour budget and hyperparameter search as our denoiser. Each is then tested on held-out operator families. Beating them means lower CER and WER on held-out operators. On TDD itself we expect only to come close to their numbers, and we will report honestly if we lose there.

The oracle. An end-to-end restorer retrained on the test operator gives an upper bound on what retraining buys. This tells us how much of the gap the modular approach recovers without retraining.

The prior-isolating ablations. These keep the PnP loop and operator identical and change only the denoiser. The variants are natural-image DRUNet as released, DRUNet retrained on documents with pixel loss only ($\lambda = 0$), our recognition-aware denoiser, and the analytical L0 prior of Deblurring Text Images via L0-Regularized Intensity and Gradient Prior inside the same loop. The recognition-aware denoiser must beat the $\lambda = 0$ version on the held-out recognizers. Otherwise the core ingredient adds nothing.

The latent-feature baseline. On the multimodal question-answering subset only, we include Uni-DocRobust. It reports corrupted-ChartQA accuracy rising from 35.71 to 59.45 for InternVL3.5-2B. We feed our restored images to the same multimodal model and compare at matched adaptation compute.

Why not just…?

Why not just retrain DRUNet on document images with ordinary pixel loss? This is the most direct threat. If it matches our method, then the contribution reduces to "use document training data," and the recognition loss is unnecessary. The $\lambda = 0$ ablation tests exactly this. We will treat a gap of under about half a point of CER on held-out recognizers as a negative result for the core claim. The motivation for expecting a gap is the evidence that pixel metrics and OCR diverge, as shown by the image-versus-OCR reporting in DocWaveDiff. That evidence comes from feed-forward models, so it is suggestive rather than conclusive for PnP.

Why not just train one end-to-end restorer on a very broad randomized mix of degradations? This is a serious threat. Domain randomization might cover enough operators that modularity stops mattering. The held-out-operator split settles it. We will train the strongest end-to-end baseline on every family except the test family, including combinations, and compare. If it reaches the PnP error rate on spatially varying blur and noisy bleed-through mixes, the case for PnP in documents weakens to runtime and interpretability. We would state that plainly.

Why not just fine-tune the OCR engine on degraded text, or correct its output afterwards? Retraining the recognizer on degraded images is a fair competitor for the recognizer it targets. However, it produces no restored image for other engines, for layout tools or for human archivists. Text-level correction also cannot recover characters that were never read, a point illustrated by the residual F1 loss at low CER in Building OCR/NER Test Collections. We include a recognizer fine-tuned on degraded data as a baseline. We also test whether running our restoration first still helps that fine-tuned recognizer, because the two approaches can be combined.

Why not put the recognizer's gradient directly into each PnP iteration at test time? That approach would steer each specific image toward the recognizer's beliefs. It is expensive, and it is especially prone to fooling a single network. We run it as a costed variant with Tesseract as the external judge. We expect it to overfit to the guiding recognizer. Training the legibility prior once amortizes that risk away.

Proposed Experiments

Experiment 1: in-distribution sanity check. On TDD and DIBCO 2017, we report PSNR, F-measure, CER and WER for every baseline. The goal is to show that the PnP pipeline is competitive where end-to-end models are strongest.

Experiment 2: the decisive test of the core claim. On the four synthetic operator families, held out in turn, we report CER and WER from Tesseract and from the held-out recognizer. Success means that recognition-aware PnP beats the fairly retrained end-to-end baselines and the $\lambda = 0$ ablation on at least three of the four held-out families, with confidence intervals obtained by bootstrapping over pages.

Experiment 3: downstream language task. On degraded Renmin pages, we restore, run OCR, and score NER F1. The anchor values are clean transcript F1 of 68.51 and original OCR F1 of 64.47. The question is whether recognition-aware restoration closes more of that gap than pixel-trained restoration at equal CER, which would indicate that it removes the kinds of errors that break word segmentation and named entities.

Experiment 4: real-data transfer check. On Old Bailey and Trove, we restore pages with an estimated blur and noise model and then run Tesseract and ABBYY, comparing against the reported WERs.

Experiment 5: noisy bleed-through. We extend the synthetic mixtures of Blind image separation for document restoration using plug-and-play approach with additive noise and report recto CER. That paper reported no OCR numbers, so this is new ground. The comparison is between its classical denoisers and ours inside the same separation loop.

Experiment 6: stability. We track how much the image changes between successive iterations, and how CER changes, as the iteration count grows, for both the unconstrained and the spectrally normalized denoiser. This quantifies how much legibility is lost to the convergence constraint that Plug-and-Play Methods Provably Converge with Properly Trained Denoisers requires.