DREAM: Document Reconstruction via End-to-end Autoregressive Model

Xin Li, Mingming Gong, Yunfei Wu, Jianxin Dai, Antai Guo, Xinghua Jiang, Haoyu Cao, Yinsong Liu, Deqiang Jiang, Xing Sun

Introduction

Document reconstruction—the process of digitizing document images into structured, machine-readable formats—is a cornerstone of modern information retrieval. Traditionally, this task has relied on complex multi-stage pipelines involving separate models for layout detection, optical character recognition (OCR), table parsing, and heuristic-based stitching. These pipelines are notoriously fragile; errors in early stages, such as misidentifying a layout element, cascade through the system, leading to poor final results.

While recent end-to-end generative models like Nougat have attempted to simplify this by directly translating images to Markdown, they often sacrifice physical layout information (bounding boxes) and suffer from slow inference speeds due to token-by-token generation of long documents. This paper introduces DREAM, an end-to-end autoregressive model designed to achieve both high-fidelity physical reconstruction and efficient logical content generation.

Methodology: The DREAM Architecture

DREAM standardizes document output as a sequence of structured elements. Each element is defined by its category, a precise bounding box, and its corresponding logical content (e.g., LaTeX for formulas, HTML for tables, or plain text for paragraphs).

The DREAM architecture featuring an Image Encoder, Feature Aggregator, and a novel Parallel Decoder for efficient document reconstruction.

The architecture is composed of three primary modules:

  1. Image Encoder: A Swin Transformer that processes the input document image into visual patch embeddings.
  2. Feature Aggregator (FA): Utilizing $N$ learnable "element queries," this module employs cross-attention to localize and aggregate features for specific document components, similar to the DETR object detection framework.
  3. Parallel Decoder: This is the core innovation of DREAM. Unlike standard autoregressive models that generate a single long string, DREAM decodes elements in parallel. It first predicts the category and bounding box for all queries. Using bipartite matching, it filters out redundant or low-confidence queries. Finally, it generates text content only for valid elements, significantly reducing the computational burden.

Evaluation and Benchmarking

To rigorously evaluate document reconstruction, the authors introduced DocRec1K, a new benchmark comprising 1,000 images from DocLayNet. Each image is meticulously annotated with layout order, bounding boxes, and formatted text. The model was also pre-trained on a massive synthetic dataset of 10 million image-text pairs derived from arXiv LaTeX sources.

Furthermore, the authors proposed the Document Similarity Metric (DSM). This holistic metric combines Intersection over Union (IoU) for physical layout accuracy and Edit Distance for logical text accuracy, providing a unified score for the entire reconstruction process.

Key Results and Performance

DREAM demonstrates superior performance across both physical and logical reconstruction tasks. On the DocRec1K benchmark, it achieved a DSM score of 91.4%, outperforming the adapted Nougat baseline's 85.2%.

Comparison of document reconstruction methods. DREAM (bottom) successfully recovers both physical layout and logical structure, avoiding the cascading errors of multi-stage methods (top) and the layout-agnostic limitations of previous end-to-end models (middle).

Beyond accuracy, DREAM offers a significant efficiency advantage. The parallel decoding mechanism allows for a 4x inference speedup over Nougat (3.2s vs. 14.1s per image), despite DREAM having fewer parameters (270M vs. 350M). The model also remains highly competitive on specialized subtasks, achieving an 89.1% F1-score on text recognition and over 98% F1 on table structure recognition benchmarks like SciTSR.

Conclusion

DREAM represents a significant step forward in document digitization by reconciling the accuracy of physical layout detection with the flexibility of end-to-end generative modeling. By treating document elements as parallelizable units, it overcomes the latency and error-propagation issues inherent in previous approaches. While current limitations include performance on "in-the-wild" photos with poor lighting, future iterations focusing on diverse training data and further decoding optimizations promise to make DREAM a robust tool for large-scale document processing.

Summary updated

BibTeX
@inproceedings{li2025dream,
  author = {Xin Li and Mingming Gong and Yunfei Wu and Jianxin Dai and Antai Guo and Xinghua Jiang and Haoyu Cao and Yinsong Liu and Deqiang Jiang and Xing Sun},
  title = {{DREAM: Document Reconstruction via End-to-end Autoregressive Model}},
  booktitle = {ACM Multimedia},
  year = {2025},
  doi = {10.1145/3746027.3754906},
}
Figure