Evaluating Menu OCR and Translation: A Benchmark for Aligning Human and Automated Evaluations in Large Vision-Language Models

Zhanglin Wu, Tengfei Song, Ning Xie, Mengli Zhu, Weidong Zhang, Shuang Wu, Pengfei Li, Chong Li, Junhao Zhu, Hao Yang, Shiliang SunarXiv 2025 · DOI · arXiv

Introduction

Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in general visual understanding. However, their performance on specialized, high-density text tasks like menu reading remains under-explored. Existing OCR benchmarks often focus on "toy" tasks—reading isolated lines or answering localized questions. To address this, researchers introduce MOTBench, a benchmark designed for the complex, real-world task of menu OCR and translation.

MOTBench focuses on long-text responses and entire information estimation compared to existing benchmarks.

The Challenge of Menu Understanding

Menus serve as a rigorous "stress test" for LVLMs due to their non-linear layouts, diverse fonts (including decorative and handwritten styles), and the necessity of precise visual grounding to associate prices with specific items. Furthermore, menu translation requires more than literal word substitution; it demands cultural context and linguistic nuance to accurately convey dish names across languages. MOTBench shifts the focus from local information estimation to entire information estimation in complex layouts.

Methodology

The MOTBench framework employs a specialized evaluation pipeline to ensure granular accuracy:

  1. Item-by-Item Evaluation: Instead of evaluating bulk text, the framework parses each menu item individually.
  2. Dual OCR Metrics: Performance is measured via acc1 (dish name recognition) and acc2 (dish name plus price and unit).
  3. LLM-Assisted Extraction: To handle the free-form nature of LVLM outputs, a secondary model (Llama-3.1-8B-Instruct) structures the responses for precise comparison against ground truth.
  4. Human Alignment: Automated scores were validated against professional human evaluations to ensure real-world relevance.

The MOTBench Dataset

The dataset comprises 130 curated menu images (65 Chinese, 65 English) across four difficulty levels:

  • Simple Layout: Standard printed documents.
  • Text-Image Mixed: Text overlaid on complex food photography.
  • Irregular Font: Stylized or handwritten text.
  • Real-World: Photos of menus in situ, such as chalkboards or outdoor signs.
Examples of the four menu categories in MOTBench, ranging from simple layouts to real-world scenes.

The dataset includes 2,328 Chinese and 1,467 English dish items, all annotated by language experts.

Key Insights

  • Cross-Lingual Disparity: Proficiency in English OCR does not guarantee success in Chinese OCR, highlighting the need for more robust multilingual training.
  • Perception vs. Reasoning: Translation remains significantly more difficult than OCR. Models often "see" the text correctly but fail to provide culturally accurate translations.
  • Spatial Reasoning: Performance declines as layouts become more irregular, indicating that models still struggle to associate spatially distant elements like a dish name and its price.

Results and Performance

Evaluation of 24 models (17 open-source, 7 closed-source) revealed:

  • Leading Models: Gemini 2.0 Flash and GPT-4.5 performed best, particularly in translation.
  • Open-Source Progress: InternVL2.5-8B-MPO and Qwen2.5-VL-7B-Instruct showed competitive OCR accuracy, rivaling closed-source counterparts.
  • Reliability: The automated evaluation method achieved over 90% consistency with human expert judgment, proving MOTBench is a reliable proxy for human evaluation.

Conclusion and Future Directions

MOTBench provides a robust framework for evaluating document-level understanding in LVLMs. Future work will aim to expand the language scope beyond English and Chinese and explore more complex downstream reasoning tasks, such as dietary recommendation based on parsed menu data.

Summary updated

Abstract

The rapid advancement of large vision-language models (LVLMs) has significantly propelled applications in document understanding, particularly in optical character recognition (OCR) and multilingual translation. However, current evaluations of LVLMs, like the widely used OCRBench, mainly focus on verifying the correctness of their short-text responses and long-text responses with simple layout, while the evaluation of their ability to understand long texts with complex layout design is highly significant but largely overlooked. In this paper, we propose Menu OCR and Translation Benchmark (MOTBench), a specialized evaluation framework emphasizing the pivotal role of menu translation in cross-cultural communication. MOTBench requires LVLMs to accurately recognize and translate each dish, along with its price and unit items on a menu, providing a comprehensive assessment of their visual understanding and language processing capabilities. Our benchmark is comprised of a collection of Chinese and English menus, characterized by intricate layouts, a variety of fonts, and culturally specific elements across different languages, along with precise human annotations. Experiments show that our automatic evaluation results are highly consistent with professional human evaluation. We evaluate a range of publicly available state-of-the-art LVLMs, and through analyzing their output to identify the strengths and weaknesses in their performance, offering valuable insights to guide future advancements in LVLM development. MOTBench is available at https://github.com/gitwzl/MOTBench.
BibTeX
@misc{wu2025evaluating,
  author = {Zhanglin Wu and Tengfei Song and Ning Xie and Mengli Zhu and Weidong Zhang and Shuang Wu and Pengfei Li and Chong Li and Junhao Zhu and Hao Yang and Shiliang Sun},
  title = {{Evaluating Menu OCR and Translation: A Benchmark for Aligning Human and Automated Evaluations in Large Vision-Language Models}},
  year = {2025},
  eprint = {2504.13945},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.13945},
}
Figure