Abstract
Text image machine translation (TIMT) is a task that translates source texts embedded in the image to target translations. The existing TIMT task mainly focuses on text-line-level images. In this paper, we extend the current TIMT task and propose a novel task, Document Image Machine Translation to Markdown (DIMT2Markdown), which aims to translate a source document image with long context and complex layout structure to markdown-formatted target translation. We also introduce a novel framework, Document Image Machine Translation with Dynamic multi-pre-trained models Assembling (DIMTDA). A dynamic model assembler is used to integrate multiple pre-trained models to enhance the model’s understanding of layout and translation capabilities. Moreover, we build a novel large-scale Document image machine Translation of ArXiv articles in markdown format (DoTA), containing 126K image-translation pairs. Extensive experiments demonstrate the feasibility of end-to-end translation of rich-text document images and the effectiveness of DIMTDA.
Introduction: From Scene Text to Full Documents
Traditional Text Image Machine Translation (TIMT) has primarily focused on "scene text," such as translating street signs or billboards. While existing end-to-end models perform well on these short, isolated text lines, they struggle significantly when faced with full academic papers or magazine pages.
The transition to document-level translation introduces three major challenges:
- Long Context: Document images contain hundreds or thousands of tokens, far exceeding the typical ~50 tokens in scene text.
- Complex Layout: Documents utilize titles, tables, figures, and formulas to define structure and meaning.
- Data Scarcity: There has been a lack of large-scale, public datasets for end-to-end document image translation.
Existing models like PEIT, which achieve 47.20 BLEU on text lines, drop to a mere 5.81 BLEU when processing full document pages, highlighting the need for a specialized approach.
Methodology: The DIMTDA Framework
To address these challenges, the authors developed DIMTDA (Document Image Machine Translation with Dynamic multi-pre-trained models Assembling). Rather than training a single model from scratch, DIMTDA assembles the strengths of specialized, pre-trained components.
The framework consists of:
- OCR Encoder: A Swin Transformer (initialized from Nougat) captures raw textual information.
- Layout Encoder: A Vision Transformer (ViT, initialized from DiT) understands global structure and element positioning.
- Dynamic Model Assembler: This core component uses two Qformer modules to unify features and a soft gate mechanism. This allows the model to dynamically weight textual versus layout information for different document regions.
- Translation Decoder: A Transformer-based decoder generates the final translation in Markdown format.
By outputting Markdown, the model represents structural elements like headers and lists directly, avoiding the need for complex rendering engines during inference.
The DoTA Dataset
The researchers introduced DoTA (Document image machine Translation of ArXiv articles), a large-scale dataset comprising 126,000 image-translation pairs. Sourced from arXiv papers (2020–2023) across various scientific fields, it provides English document images paired with Chinese Markdown translations. Quality was ensured using the COMET metric and professional validation.
Key Results and Insights
DIMTDA demonstrates a significant performance leap, achieving a BLEU score of 38.68 on the full test set, compared to the 1.5–5.8 range of previous models. It also outperformed commercial MLLMs like GPT-4V and Gemini in structural consistency and Markdown formatting.
Key findings include:
- Layout Importance: Ablation studies confirmed that layout encoding is essential for understanding reading order and context in complex documents.
- Specialized Initialization: Initializing the OCR encoder with document-specific models (like Nougat) proved far more effective than general-purpose OCR pre-training.
- End-to-End Efficiency: DIMTDA avoids the error propagation common in "cascade" systems (OCR → Layout → Translation) and reduces inference time by approximately 21%.
Conclusion and Future Work
While DIMTDA represents a breakthrough in document-level translation, challenges remain. The model still faces difficulties with extremely dense numerical data in formulas and complex table structures. Technical abbreviations can also lead to occasional misreadings. Future work will focus on more explicit modeling of mathematical formulas and tables to better handle the information density found in scientific literature.
Reference: Liang, Y., Zhang, Y., et al. (2024). Document Image Machine Translation with Dynamic Multi-pre-trained Models Assembling. Proceedings of NAACL 2024.
Summary updated