Introduction
Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities in general image understanding, yet they often struggle with dense, multi-page documents. These documents require a difficult balance: maintaining high resolution for text recognition while efficiently processing natural image features. Current approaches typically face a "token wall," where high-resolution patching leads to computationally prohibitive token counts for multi-page contexts.
Fox is a novel pipeline and training strategy designed to overcome these limitations. It enables LVLMs to perform fine-grained tasks—such as region-specific OCR, translation, and summarization—across single or multi-page documents. By "catalyzing" multiple vision vocabularies with hybrid data, Fox achieves state-of-the-art performance on dense document understanding using an efficient 1.8B parameter model.
The Challenge: Vocabulary Bias and Context Constraints
Modern LVLMs generally fall into two categories, both with significant drawbacks for document processing:
- Patch-based models: These crop images into small tiles to preserve resolution. However, this generates thousands of tokens per page, making multi-page processing memory-intensive and slow.
- Global encoding models: These compress pages into fewer tokens but often suffer from "vocabulary bias," where the model favors natural image features (CLIP-style) over document-specific features (Doc-style), or vice versa.
Furthermore, most models lack the ability for fine-grained interaction, such as responding to a user request to "OCR just this specific red box" or "Summarize the third paragraph on page 5."
Methodology: Dual-Branch Architecture and Catalyzation
The Fox architecture utilizes a dual-branch vision system to capture the full spectrum of document content.
Cross-Vocabulary Catalyzation
Fox employs a CLIP-style branch for natural visual elements and a Doc-style branch (based on SAM/Vary-tiny) for dense text and charts. To prevent the model from ignoring one branch in favor of the other, the researchers introduced "catalyzer" data: synthetic images where natural figures are rendered directly onto PDF pages. This forces the model to utilize both visual vocabularies simultaneously to describe interleaved content.
Position-Aware Instructions
Fox is trained to interpret precise coordinates (points and bounding boxes) and visual cues like color-coded boxes. This allows the model to act as a "digital reading pen," focusing on specific user-defined regions for targeted tasks like line-level OCR or paragraph translation.
Multi-page Unification
By achieving a high compression ratio—representing a 1024x1024 page with only 256 tokens—Fox can concatenate up to 8 pages into a single conversation window. This efficiency enables cross-page reasoning, such as comparing content across different sections of a long document.
Data and Benchmarks
The model was trained on a massive dataset including 4.6M document image-text pairs, 1M natural images, and 800K multi-page samples. To evaluate its unique capabilities, the authors developed a bilingual (English/Chinese) benchmark consisting of 9 sub-tasks, ranging from region-level OCR to multi-page Visual Question Answering (VQA).
Key Results
Despite its compact 1.8B parameter size, Fox outperforms much larger models in dense document tasks:
- Dense OCR: On English document recognition, Fox achieved an Edit Distance of 0.046, significantly outperforming LLaVA-NeXT-34B (0.430) and InternVL-Chat-V1.5-26B (0.393).
- Multi-page Performance: In 8-page document tasks, Fox achieved a 94.6% F1-score for multi-region OCR and 82.7% accuracy on cross-page VQA.
- Versatility: Qualitative results demonstrate Fox successfully handling complex layouts, academic papers with embedded figures, and children's storybooks.
Conclusion and Future Work
Fox demonstrates that efficient, fine-grained document understanding is possible through dual-branch vocabularies and position-aware training. While the model excels at multi-page OCR and region-specific tasks, future work will focus on improving high-level reasoning across multi-page contexts and increasing the resolution of the CLIP branch to capture even finer details within embedded natural images.
Summary updated