TL;DR
This paper introduces Nayana, a novel framework for adapting Vision-Language Models (VLMs) to low-resource languages for document OCR. By leveraging a layout-aware synthetic data generation pipeline and parameter-efficient adaptation (LoRA), Nayana achieves competitive performance without requiring extensive manual annotations, opening doors for AI accessibility in underserved linguistic communities.
What Problem Does This Paper Address?
Modern Vision-Language Models (VLMs) have shown remarkable success in high-resource languages like English, particularly in tasks like Optical Character Recognition (OCR) and document understanding. However, their global applicability is severely limited by the scarcity of high-quality training data in low-resource languages. Creating these datasets is time-consuming and expensive, especially for languages with complex scripts. This paper tackles this fundamental challenge by proposing a scalable and efficient framework, Nayana, to adapt existing VLMs to new languages using purely synthetic data.
Methods
Nayana's core innovation lies in its two-pronged approach:
- Layout-Aware Synthetic Data Generation: Instead of relying on manual annotations, Nayana generates synthetic training data. It starts with English documents, processes them through a pipeline that includes layout analysis (using DocLayout-YOLO), high-resolution OCR (using Tesseract), and context-aware translation. Crucially, this pipeline preserves the original document's layout, visual hierarchy, and structural relationships, creating realistic multilingual datasets.
- Parameter-Efficient Adaptation (LoRA): To adapt existing VLMs to these synthetic datasets, Nayana employs Low-Rank Adaptation (LoRA). This technique allows for efficient parameter updates, enabling models to learn new languages with minimal computational overhead and without catastrophic forgetting of their original capabilities. The paper systematically analyzes different LoRA configurations to find optimal settings for multilingual adaptation.
The overall architecture involves a multi-stage processing pipeline that includes document preprocessing, layout analysis, text extraction, visual attribute analysis, and translation, all while maintaining document integrity.
Datasets
- Seed Dataset: The synthetic data generation pipeline is seeded with approximately 14,000 English-language documents from diverse sources: arXiv (research papers), PubMed (medical literature), newspapers, and marketing materials.
- Evaluation Dataset: A carefully balanced test set of 500 images per language was constructed, drawing 40% from arXiv, 40% from PubMed, and 20% from newspaper and advertising materials. This dataset was translated into parallel versions across all ten target languages.
Key Insights and Ideas
- Synthetic Data is Sufficient: The paper strongly demonstrates that high-quality synthetic data, when generated with attention to layout and structure, can be sufficient for training VLMs for OCR in new languages, significantly reducing the reliance on expensive manual annotations.
- Layout Preservation is Crucial: Preserving the visual and structural integrity of documents during synthetic data generation is key to achieving good performance. Simply translating text without considering layout leads to poor results.
- LoRA is Effective for Multilingual Adaptation: Parameter-efficient fine-tuning with LoRA proves to be a robust method for adapting VLMs to new languages, maintaining performance across multiple scripts and avoiding catastrophic forgetting.
- Cross-Script Learning is Possible: The research shows that models adapted to one Indic script can exhibit improved performance on related scripts (e.g., Hindi to Marathi), highlighting the potential for beneficial cross-script transfer.
- Vocabulary Expansion is Not the Primary Hurdle: Initial experiments with expanding model vocabulary for Indic scripts showed poor convergence, suggesting that the core challenge lies in the visual processing of complex scripts rather than just vocabulary representation.
Key Results
- Nayana-OCR achieved competitive character-level accuracy (CER) and significantly outperformed baselines in semantic metrics (BLEU) across ten Indic languages.
- For Hindi, the optimal LoRA configuration (r=64, α=128) achieved a BLEU score of 0.58, a substantial improvement over the baseline LoRA (0.29).
- Nayana-OCR achieved an average CER of 0.227 and an average BLEU of 0.395 across all evaluated languages, outperforming traditional OCR systems like Tesseract in semantic understanding.
- The framework processes individual documents in approximately 3-5 seconds, enabling rapid dataset generation.
- The synthetic data generation pipeline achieves a 7-10x image multiplication factor per source document.
- Nayana-OCR demonstrated a 76% reduction in Character Error Rate compared to the base GOT OCR model.
Limitations and Future Work
- Inference Latency: Nayana-OCR exhibits higher inference latency compared to traditional OCR systems, reflecting the complexity of its vision-language processing.
- Script-Specific Performance Variations: Performance variations across different scripts suggest room for improvement in handling certain complex writing systems.
- Synthetic Data Realism: While effective, the synthetic data generation might not capture all real-world variations in document layouts and styles.
Future work includes expanding the diversity of seed datasets, incorporating more complex document structures, developing specialized architectures for better performance/efficiency balance, and exploring Nayana's application to other vision-language tasks. The authors also plan to release open-source tools to facilitate broader adoption.
Summary updated