Introduction
Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in general visual understanding. However, their performance on specialized, high-density text tasks like menu reading remains under-explored. Existing OCR benchmarks often focus on "toy" tasks—reading isolated lines or answering localized questions. To address this, researchers introduce MOTBench, a benchmark designed for the complex, real-world task of menu OCR and translation.
The Challenge of Menu Understanding
Menus serve as a rigorous "stress test" for LVLMs due to their non-linear layouts, diverse fonts (including decorative and handwritten styles), and the necessity of precise visual grounding to associate prices with specific items. Furthermore, menu translation requires more than literal word substitution; it demands cultural context and linguistic nuance to accurately convey dish names across languages. MOTBench shifts the focus from local information estimation to entire information estimation in complex layouts.
Methodology
The MOTBench framework employs a specialized evaluation pipeline to ensure granular accuracy:
- Item-by-Item Evaluation: Instead of evaluating bulk text, the framework parses each menu item individually.
- Dual OCR Metrics: Performance is measured via
acc1(dish name recognition) andacc2(dish name plus price and unit). - LLM-Assisted Extraction: To handle the free-form nature of LVLM outputs, a secondary model (Llama-3.1-8B-Instruct) structures the responses for precise comparison against ground truth.
- Human Alignment: Automated scores were validated against professional human evaluations to ensure real-world relevance.
The MOTBench Dataset
The dataset comprises 130 curated menu images (65 Chinese, 65 English) across four difficulty levels:
- Simple Layout: Standard printed documents.
- Text-Image Mixed: Text overlaid on complex food photography.
- Irregular Font: Stylized or handwritten text.
- Real-World: Photos of menus in situ, such as chalkboards or outdoor signs.
The dataset includes 2,328 Chinese and 1,467 English dish items, all annotated by language experts.
Key Insights
- Cross-Lingual Disparity: Proficiency in English OCR does not guarantee success in Chinese OCR, highlighting the need for more robust multilingual training.
- Perception vs. Reasoning: Translation remains significantly more difficult than OCR. Models often "see" the text correctly but fail to provide culturally accurate translations.
- Spatial Reasoning: Performance declines as layouts become more irregular, indicating that models still struggle to associate spatially distant elements like a dish name and its price.
Results and Performance
Evaluation of 24 models (17 open-source, 7 closed-source) revealed:
- Leading Models: Gemini 2.0 Flash and GPT-4.5 performed best, particularly in translation.
- Open-Source Progress: InternVL2.5-8B-MPO and Qwen2.5-VL-7B-Instruct showed competitive OCR accuracy, rivaling closed-source counterparts.
- Reliability: The automated evaluation method achieved over 90% consistency with human expert judgment, proving MOTBench is a reliable proxy for human evaluation.
Conclusion and Future Directions
MOTBench provides a robust framework for evaluating document-level understanding in LVLMs. Future work will aim to expand the language scope beyond English and Chinese and explore more complex downstream reasoning tasks, such as dietary recommendation based on parsed menu data.
Summary updated