Abstract
TextDoctor is a unified document inpainting framework designed to restore corrupted text by mimicking human reading behavior—learning fundamental text features from small patches and applying them across high-resolution images. By combining multi-scale structure prediction with patch-based diffusion models, it achieves state-of-the-art restoration quality and generalizes to unseen document styles, such as checks, forms, and social media posts, without requiring domain-specific fine-tuning.
Introduction
Restoring text in degraded documents—whether due to environmental damage, poor scanning, or human interference—is a complex task. Existing state-of-the-art (SOTA) methods face two major hurdles: a generalization gap and the high-resolution memory wall. Most models are "specialists," failing when applied to document layouts or textures outside their training set. Furthermore, processing modern high-resolution scans (2K to 8K) often leads to out-of-memory (OOM) errors on standard GPUs. TextDoctor addresses these challenges by creating a "generalist" restorer that handles ultra-high resolutions efficiently.
Methodology
The core philosophy of TextDoctor is inspired by human reading behavior: recognizing fine-grained letters and words before grasping broader context. The architecture consists of two main stages:
Patch-based Structure Pyramid Prediction (PSPP)
Before restoring pixels, the model predicts a binary structure map of where text should be. To handle varying text sizes, the input is resized into an image pyramid. TextDoctor processes these scales using overlapping patches and fuses them into a single, high-resolution structure map, ensuring both tiny footnotes and large headers are captured accurately.
Conditional Patch Pyramid Denoising (CPPD)
The restoration is handled by a Diffusion Model using "Pyramid Patches." It partitions the corrupted image and predicted structure map into patches of different sizes (e.g., $128^2$ and $256^2$). The model is trained to recover clean text patches conditioned on the corrupted input and the predicted structure. By using multiple patch sizes, the model captures both local fine details and broader strokes, which are merged back into the full-resolution document.
Key Insights: Text is Universal
A central insight of this work is that while document layouts vary wildly, the visual features of text elements (strokes, curves, fonts) are remarkably similar across domains. By training on universal text image patches rather than specific document types, the model learns "how to draw text" rather than "how a specific document looks." Additionally, using the predicted structure map as a prior provides a "skeleton" for the diffusion process, significantly improving legibility and preventing hallucinations.
Key Results
TextDoctor was evaluated on seven diverse public datasets, including FUNSD (forms), BCSD (bank checks), and subsets of Roboflow 100 (tweets, diagrams, tables).
- Superior Accuracy: On the FUNSD dataset, TextDoctor achieved a Word Accuracy (PaddleOCR) of 55.38%, a significant improvement over the previous best method, DocDiff (45.24%).
- Zero-Shot Generalization: The model outperformed SOTA competitors across all tested datasets without domain-specific fine-tuning.
- Resolution Scalability: While competitors suffered OOM errors at $2048^2$ or $4096^2$ resolutions, TextDoctor successfully processed images up to 8K ($8192^2$).
- Visual Quality: Qualitative results demonstrate sharper text edges and more accurate character reconstructions compared to GAN-based or standard Transformer-based inpainters.
Conclusion and Future Work
TextDoctor provides a robust, unified framework for document inpainting. While highly effective at restoring strokes, it faces challenges with "large masks" where entire words are missing, potentially requiring linguistic context from Large Language Models (LLMs) for semantic reconstruction. Future work will explore integrating such semantic guidance and optimizing inference speed for real-time applications.