TL;DR
This paper introduces BigDocs-7.5M, a massive, license-permissive dataset for training multimodal models on document and code tasks. It also presents BigDocs-Bench, a benchmark suite with 10 novel tasks, and demonstrates that models trained on BigDocs significantly outperform existing models, including closed-source ones like GPT-4, on these challenging document understanding and code generation tasks.
What Problem Does This Paper Address?
The paper addresses a critical gap in multimodal AI research: the lack of large-scale, open-access, and permissively licensed datasets for training models on complex document understanding and code generation tasks. While existing multimodal models show promise, their commercial applications are often hindered by proprietary datasets with restrictive licenses. Current open-source datasets tend to focus on simpler tasks like basic OCR or question-answering, failing to capture the complexity of visually-rich documents (e.g., charts, diagrams, code snippets) or the generation of long, structured outputs like HTML or LaTeX. This limits the advancement of open-source multimodal AI for real-world document-centric applications.
Methods
The authors introduce a rigorous data curation process for BigDocs-7.5M, emphasizing accountability, responsibility, and transparency. This involves:
- License Filtering: Prioritizing datasets with permissive licenses (CC-BY-4.0, MIT) and excluding those with non-commercial or unclear licensing.
- Data Curation: A multi-stage process including OCR extraction, HTML/SVG/Markdown curation, content filtering, VQA formatting, and text-image alignment.
- BigDocs Toolkit: A modular framework for data preprocessing, filtering, contamination management, and metadata management, including a unified metadata framework for enhanced traceability.
- BigDocs-Bench: A suite of 10 novel downstream tasks designed to evaluate multimodal models on structured output generation (HTML, LaTeX, SVG, Markdown) and reasoning over GUIs. These tasks are crucial for real-world applications.
- Training Strategy: A three-stage training pipeline: (1) General Vision-Language Pretraining (relying on public checkpoints), (2) Document-Specific Continual Pretraining (using BigDocs-7.5M), and (3) Supervised Finetuning (using BigDocs-Bench and other downstream datasets).
Datasets
- BigDocs-7.5M: A new, large-scale, license-permissive dataset comprising 7.5 million image-text pairs, curated from 16 public datasets and crawled sources. It supports tasks in Document Information Extraction, Document Understanding, and Document Creation and Manipulation.
- BigDocs-Bench: A benchmark suite with 10 novel downstream tasks, including:
- Screenshot2HTML
- Table2LaTeX
- Image2SVG
- Image2Flow (GraphViz and JSON)
- Chart2Markdown
- Chart2Caption
- GUI2UserIntent
- GUI2Summary
- GUI-VQA These tasks focus on generating structured outputs from visual inputs and reasoning over GUIs, aiming to reflect real-world use cases.
Key Insights and Ideas
- License Permissiveness is Crucial: The paper strongly emphasizes the importance of permissive licensing for fostering open-source AI development and commercial adoption. Their meticulous filtering process highlights the prevalence of restrictive licenses in existing datasets.
- BigDocs Outperforms Existing Datasets: Models trained on BigDocs consistently show improved performance across various general document benchmarks and the novel BigDocs-Bench tasks, even outperforming models trained on larger, but less permissively licensed, datasets like DocStruct4M.
- VQA Formatting is Key for Instruction Following: Ablation studies reveal that including VQA formatting during continual pretraining significantly improves a model's ability to adhere to requested output formats and generalize to downstream tasks.
- OCR Data Enhances Precision: Removing OCR tasks from the training data leads to increased hallucination and decreased precision in information extraction tasks, underscoring the value of robust OCR capabilities.
- Context Length is Not the Sole Bottleneck for Complex Tasks: For tasks like Screenshot2HTML, where output length can be substantial, simply increasing context length did not yield significant performance gains, suggesting other factors like data quality and model architecture are more critical.
- BigDocs-Bench Tasks are Unique: Correlation analysis shows that BigDocs-Bench tasks are distinct from existing benchmarks, covering unique aspects of document understanding and code generation not adequately addressed elsewhere.
Key Results
- Performance Improvement: Models trained with BigDocs-Bench achieved up to a 25.8% average performance improvement over closed-source GPT-40 on document reasoning and structured output tasks.
- Human Evaluation Preference: Human evaluations showed a preference for outputs from models trained on BigDocs over GPT-40 in several tasks. For example, in Table2LaTeX, Phi3.5 BigDocs won 63% of the time against GPT-40.
- State-of-the-Art Performance: BigDocs models (e.g., Phi3.5-V-4B + BigDocs) achieved top performance on 8 out of 10 BigDocs-Bench tasks, demonstrating their strong capabilities in generating structured code outputs and reasoning from GUIs.
- Reduced Contamination: BigDocs-7.5M exhibits lower data contamination rates compared to DocStruct4M on several benchmarks, as assessed by cosine similarity metrics.
Limitations and Future Work
- Suboptimal Performance on Some Benchmarks: The authors acknowledge that BigDocs models still show suboptimal performance on some public benchmarks, suggesting a need to refine the data mixture and explore additional data sources.
- Limited Context Length: Models are currently trained with a maximum context length of 8192 tokens, which can restrict performance on tasks requiring very long, structured outputs like HTML and SVG.
- Commercial Viability Uncertainty: The paper notes uncertainty regarding the commercial viability of base models due to a lack of transparency in their pretraining data.
- Future Work: The authors suggest refining the data mixture, exploring longer context lengths, and developing more robust evaluation metrics for complex document understanding tasks. They also highlight the potential for BigDocs to facilitate research into multimodal models for code generation and document reasoning.
Summary updated