Focus Anywhere for Fine-grained Multi-page Document Understanding

Chenglong Liu, Haoran Wei, Jinyue Chen, Lingyu Kong, Zheng Ge, Zining Zhu, Liang Zhao, Jianjian Sun, Chunrui Han, Xiangyu ZhangarXiv 2024 · DOI · arXiv

Introduction

Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities in general image understanding, yet they often struggle with dense, multi-page documents. These documents require a difficult balance: maintaining high resolution for text recognition while efficiently processing natural image features. Current approaches typically face a "token wall," where high-resolution patching leads to computationally prohibitive token counts for multi-page contexts.

Overview of Fox capabilities including cross-vocabulary pairs, position-aware instructions, and multi-page support.

Fox is a novel pipeline and training strategy designed to overcome these limitations. It enables LVLMs to perform fine-grained tasks—such as region-specific OCR, translation, and summarization—across single or multi-page documents. By "catalyzing" multiple vision vocabularies with hybrid data, Fox achieves state-of-the-art performance on dense document understanding using an efficient 1.8B parameter model.

The Challenge: Vocabulary Bias and Context Constraints

Modern LVLMs generally fall into two categories, both with significant drawbacks for document processing:

  1. Patch-based models: These crop images into small tiles to preserve resolution. However, this generates thousands of tokens per page, making multi-page processing memory-intensive and slow.
  2. Global encoding models: These compress pages into fewer tokens but often suffer from "vocabulary bias," where the model favors natural image features (CLIP-style) over document-specific features (Doc-style), or vice versa.

Furthermore, most models lack the ability for fine-grained interaction, such as responding to a user request to "OCR just this specific red box" or "Summarize the third paragraph on page 5."

Methodology: Dual-Branch Architecture and Catalyzation

The Fox architecture utilizes a dual-branch vision system to capture the full spectrum of document content.

The Fox architecture featuring dual-branch vision vocabularies and position-aware prompt processing.

Cross-Vocabulary Catalyzation

Fox employs a CLIP-style branch for natural visual elements and a Doc-style branch (based on SAM/Vary-tiny) for dense text and charts. To prevent the model from ignoring one branch in favor of the other, the researchers introduced "catalyzer" data: synthetic images where natural figures are rendered directly onto PDF pages. This forces the model to utilize both visual vocabularies simultaneously to describe interleaved content.

Position-Aware Instructions

Fox is trained to interpret precise coordinates (points and bounding boxes) and visual cues like color-coded boxes. This allows the model to act as a "digital reading pen," focusing on specific user-defined regions for targeted tasks like line-level OCR or paragraph translation.

Multi-page Unification

By achieving a high compression ratio—representing a 1024x1024 page with only 256 tokens—Fox can concatenate up to 8 pages into a single conversation window. This efficiency enables cross-page reasoning, such as comparing content across different sections of a long document.

Data and Benchmarks

The model was trained on a massive dataset including 4.6M document image-text pairs, 1M natural images, and 800K multi-page samples. To evaluate its unique capabilities, the authors developed a bilingual (English/Chinese) benchmark consisting of 9 sub-tasks, ranging from region-level OCR to multi-page Visual Question Answering (VQA).

Key Results

Despite its compact 1.8B parameter size, Fox outperforms much larger models in dense document tasks:

  • Dense OCR: On English document recognition, Fox achieved an Edit Distance of 0.046, significantly outperforming LLaVA-NeXT-34B (0.430) and InternVL-Chat-V1.5-26B (0.393).
  • Multi-page Performance: In 8-page document tasks, Fox achieved a 94.6% F1-score for multi-region OCR and 82.7% accuracy on cross-page VQA.
  • Versatility: Qualitative results demonstrate Fox successfully handling complex layouts, academic papers with embedded figures, and children's storybooks.

Conclusion and Future Work

Fox demonstrates that efficient, fine-grained document understanding is possible through dual-branch vocabularies and position-aware training. While the model excels at multi-page OCR and region-specific tasks, future work will focus on improving high-level reasoning across multi-page contexts and increasing the resolution of the CLIP branch to capture even finer details within embedded natural images.

Summary updated

Abstract

Modern LVLMs still struggle to achieve fine-grained document understanding, such as OCR/translation/caption for regions of interest to the user, tasks that require the context of the entire page, or even multiple pages. Accordingly, this paper proposes Fox, an effective pipeline, hybrid data, and tuning strategy, that catalyzes LVLMs to focus anywhere on single/multi-page documents. We introduce a novel task to boost the document understanding by making LVLMs focus attention on the document-level region, such as redefining full-page OCR as foreground focus. We employ multiple vision vocabularies to extract visual hybrid knowledge for interleaved document pages (e.g., a page containing a photo). Meanwhile, we render cross-vocabulary vision data as the catalyzer to achieve a full reaction of multiple visual vocabularies and in-document figure understanding. Further, without modifying the weights of multiple vision vocabularies, the above catalyzed fine-grained understanding capabilities can be efficiently tuned to multi-page documents, enabling the model to focus anywhere in both format-free and page-free manners. Besides, we build a benchmark including 9 fine-grained sub-tasks (e.g., region-level OCR/summary, color-guided OCR) to promote document analysis in the community. The experimental results verify the superiority of our model.
BibTeX
@misc{liu2024focus,
  author = {Chenglong Liu and Haoran Wei and Jinyue Chen and Lingyu Kong and Zheng Ge and Zining Zhu and Liang Zhao and Jianjian Sun and Chunrui Han and Xiangyu Zhang},
  title = {{Focus Anywhere for Fine-grained Multi-page Document Understanding}},
  year = {2024},
  eprint = {2405.14295},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.14295},
}
Figure