Reviving Cultural Heritage: A Novel Approach for Comprehensive Historical Document Restoration

Yuyi Zhang, Peirong Zhang, Zhenhua Yang, Pengyu Yan, Yongxin Shi, Pengwei Liu, Fengjun Guo, Lianwen Jin

TL;DR

AutoHDR is a fully automated, full-page historical document restoration framework that jointly reconstructs degraded text and visual appearance by mimicking the workflow of expert historians. Alongside this framework, the authors introduce FPHDR, the first full-page dataset for historical document restoration, enabling models to leverage broad contextual semantics and achieve state-of-the-art restoration fidelity.

What Problem Does This Paper Address?

Historical documents—such as ancient books, rubbings, and scrolls—are invaluable cultural treasures that naturally degrade over centuries due to tears, water erosion, and oxidation.

While automated Historical Document Restoration (HDR) has seen progress, existing methods suffer from three critical limitations:

  1. Single-Modality Focus: Most models restore either the text (NLP-based) or the visual appearance (vision-based), but rarely both jointly.
  2. Patch-Level Constraints: Existing approaches operate on tiny image patches or single characters. This prevents them from leveraging full-page contextual semantic information, causing them to fail when damage spans across larger regions.
  3. Lack of Automation: Current workflows address only isolated stages of restoration, requiring heavy manual intervention from human experts.

This paper introduces a holistic, full-page solution that automates the entire pipeline while allowing for seamless human-machine collaboration.

Methods

AutoHDR breaks down the restoration process into three distinct, interconnected modular stages that mirror how human historians work:

Overall workflow of the proposed AutoHDR framework
  1. OCR-Assisted Damage Localization (OADL): The system runs a character-level OCR model (YOLOv7 + ViT) alongside a specialized damage localization model (DINO). A fusion mechanism merges their outputs: characters with low OCR confidence are flagged as ambiguous, and their bounding boxes are combined with the damage detector's coordinates to map out exactly what needs fixing.
  2. Damaged Content Prediction (DCP): Missing text is predicted using a large language model (Qwen2) fine-tuned on classical Chinese corpora. To optimize this, the authors propose Vision-Language Context Prediction (VLCP). VLCP routes lightly damaged characters directly through the OCR engine and reserves the LLM's heavy linguistic processing power for severely damaged, unreadable text.
  3. Historical Appearance Restoration (HAR): A diffusion-based image-to-image model reconstructs the document at the pixel level. To scale this patch-based diffusion model to full pages, the authors introduce a Patch-AutoRegressive (PAR) mechanism. PAR dynamically selects the cleanest corner of a document to start restoring, then uses an overlapping sliding window to propagate visual consistency across the entire page.

Datasets

The authors introduce FPHDR (Full-Page Historical Document Restoration), a pioneer dataset designed to shift the community from patch-level to page-level restoration.

  • Real Samples: 1,633 real degraded pages from the Fangshan Stone Sutras (spanning nearly a millennium from the Sui to the Ming Dynasty). These were meticulously annotated by ten HDR experts with character/line locations, text content, and damage grades.
  • Synthetic Training Pairs: 6,543 high-quality synthetic full-page pairs. Because diffusion models require pixel-level ground truths (which are impossible to manually draw for degraded ancient texts), the authors applied realistic digital degradation—such as character removal (via LAMA inpainting), paper masking, and simulated ink erosion—to well-preserved historical documents.

Key Insights and Ideas

  • The Synergy of OCR and LLMs: Instead of relying solely on an LLM to guess missing text, the VLCP algorithm exploits the fact that OCR can still read lightly damaged characters. By letting OCR handle the easy parts, the LLM's contextual prediction burden is drastically reduced, minimizing hallucinated text.
  • Images as Sequences: Diffusion models struggle with full-page generation consistency. The Patch-AutoRegressive (PAR) mechanism elegantly treats full-page image restoration similarly to autoregressive language modeling—using previously restored image patches as "tokens" of context for the next patch.
  • Dual-Mode Restoration Flexibility: To the authors' surprise, AutoHDR is capable of two distinct restoration styles: "restoring the old as old" (retaining historical paper textures and aging styles) and "restoring the old as new" (rendering crisp, clean characters on a pristine background).
Restoration styles: old as old vs. old as new

Key Results

  • Downstream OCR Accuracy: On severely damaged documents where standard OCR accuracy starts at a low 46.83%, AutoHDR automatically boosts recognition accuracy to 84.05%. When historians collaborate with the system (selecting from the model's Top-5 suggestions), accuracy reaches 94.25%.
  • Text Prediction Performance: AutoHDR built on Qwen2-7B achieved a remarkable 95.15% Top-1 text accuracy when paired with the VLCP algorithm, outperforming historical text models like SikuBERT and Ithaca.
  • Visual Quality: In pixel-level evaluations on synthetic data, AutoHDR achieved the lowest LPIPS score (0.0541), beating out prominent image restoration baselines like NAFNet, Uformer, and Restormer, which frequently suffered from stroke distortion or blurry artifacts.

Limitations and Future Work

Despite its strong performance, AutoHDR has a few notable limitations:

  • Inference Speed: Because it relies on a multi-stage pipeline involving object detection, LLM inference, and iterative diffusion sampling, it takes approximately 5 minutes to process a single page on an NVIDIA A10 GPU.
  • Extreme Degradation: In cases where documents are almost entirely obliterated, the LLM can still generate plausible but historically incorrect text.

Future Work: The authors plan to explore unifying this multi-stage pipeline into a single, end-to-end large vision-language model (such as Qwen2.5-VL or InternVL 2.5) to drastically improve processing speed and structural cohesion.

Summary updated

Abstract

Historical documents represent an invaluable cultural heritage, yet have undergone significant degradation over time through tears, water erosion, and oxidation. Existing Historical Document Restoration (HDR) methods primarily focus on single modality or limited-size restoration, failing to meet practical needs. To fill this gap, we present a full-page HDR dataset (FPHDR) and a novel automated HDR solution (AutoHDR). Specifically, FPHDR comprises 1,633 real and 6,543 synthetic images with character-level and line-level locations, as well as character annotations in different damage grades. AutoHDR mimics historians' restoration workflows through a three-stage approach: OCR-assisted damage localization, vision-language context text prediction, and patch autoregressive appearance restoration. The modular architecture of AutoHDR enables seamless human-machine collaboration, allowing for flexible intervention and optimization at each restoration stage. Experiments demonstrate AutoHDR's remarkable performance in HDR. When processing severely damaged documents, our method improves OCR accuracy from 46.83% to 84.05%, with further enhancement to 94.25% through human-machine collaboration. We believe this work represents a significant advancement in automated historical document restoration and contributes substantially to cultural heritage preservation. The model and dataset are available at https://github.com/SCUT-DLVCLab/AutoHDR.
BibTeX
@inproceedings{zhang2025reviving,
  author = {Yuyi Zhang and Peirong Zhang and Zhenhua Yang and Pengyu Yan and Yongxin Shi and Pengwei Liu and Fengjun Guo and Lianwen Jin},
  title = {{Reviving Cultural Heritage: A Novel Approach for Comprehensive Historical Document Restoration}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1402},
}
Figure