KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to Diverse Visual Contexts

Taebaek Hwang, Minseo Kim, Gisang Lee, Seonuk Kim, Hyunjun Eun

Abstract

KRETA is a large-scale benchmark designed to evaluate Vision-Language Models (VLMs) on their ability to read and reason over Korean text in diverse visual contexts. By introducing a dual-level framework—distinguishing between basic recognition ("System 1") and complex reasoning ("System 2")—the researchers reveal a significant "reasoning bottleneck" where even state-of-the-art models struggle to apply logic to the text they successfully transcribe. The benchmark covers 15 industrial domains and 26 distinct image types, providing a high-quality, native Korean dataset for higher-order reasoning.

Introduction

While Vision-Language Models have made massive strides in English-centric tasks, low-resource languages like Korean face a "data desert" in text-rich Visual Question Answering (VQA). Existing Korean VQA datasets are often small, limited to simple document types, or rely on translated English questions that miss cultural nuances. Furthermore, most benchmarks conflate simple OCR (Optical Character Recognition) with actual reasoning. A model might correctly read a price tag but fail to calculate a discount or understand a complex store sign's instructions. KRETA addresses this by providing a high-quality, native Korean dataset that specifically targets higher-order reasoning across 15 industrial domains and 26 distinct image types.

Methodology

The authors developed a sophisticated semi-automated VQA generation pipeline to ensure data quality and scalability. This pipeline involves stepwise image decomposition using multiple VLMs to generate structured captions, followed by a dual-level QA generation process inspired by Kahneman’s "Thinking, Fast and Slow."

Overview of the KRETA dataset distribution and the semi-automated VQA generation pipeline.
  1. Stepwise Image Decomposition: Multiple VLMs generate structured captions describing the scene, layout, and textual relationships.
  2. Dual-Level QA Generation:
    • System 1: Basic text recognition and intuitive understanding.
  3. System 2: Advanced reasoning requiring multi-step logic, mathematical calculation, or domain-specific knowledge.
  4. Rigorous Evaluation Protocol: Candidate QA pairs are scored across seven metrics by a committee of VLMs.
  5. Hard Negative Generation: An LLM generates three plausible but incorrect options to prevent models from guessing.
  6. Human Refinement: Final manual review ensures accuracy and cultural relevance.

The KRETA Dataset

The KRETA benchmark consists of 2,577 total samples, divided into 1,426 System 1 and 1,151 System 2 questions. It spans 15 domains aligned with the Korean Standard Industrial Classification and 26 image types, ranging from test papers and mobile screenshots to store signs and handwriting. Compared to existing benchmarks like K-MMB or MTVQA-ko, KRETA offers superior scale and a dedicated focus on complex reasoning.

Experimental Results and Insights

Evaluation of various VLMs on KRETA reveals a massive performance gap between reading and reasoning, termed the "Reasoning Bottleneck." While top models achieve near-perfect scores (~98%) on System 1 tasks, their performance drops significantly to ~70% on System 2 tasks.

Qualitative examples of System 1 (recognition) and System 2 (reasoning) QA pairs across various domains in KRETA.
  • Top Performers: Gemini 2.0-flash (85.4%) and GPT-4o (84.6%) lead the pack, showing robust performance across most domains.
  • Open-Source Gap: The best open-source model, VARCO-VISION (14B), achieved 72.3%, trailing closed-source leaders by over 12 percentage points.
  • Domain Sensitivity: Models struggle with Korean-specific contexts, such as CSAT History and Science. GPT-4o excelled in CSAT History (93.3%), while smaller models like Qwen2.5-VL (7B) struggled in CSAT Science (15.1%).
  • Layout Challenges: "Scene Text" (banners, signs) remains harder than structured "Documents" (tables, receipts) due to occlusions and unconventional layouts.
  • The CoT Paradox: Chain-of-Thought (CoT) prompting helps powerful closed-source models but can degrade the performance of smaller open-source models by overwhelming their reasoning capacity.

Conclusion

KRETA highlights the critical need for benchmarks that separate recognition from reasoning in low-resource languages. Future work will look into expanding the benchmark to multi-image reasoning, refining the taxonomy of System 2 logic, and adapting the semi-automated pipeline to other low-resource languages to accelerate global VLM research.

Summary updated

BibTeX
@inproceedings{hwang2025kreta,
  author = {Taebaek Hwang and Minseo Kim and Gisang Lee and Seonuk Kim and Hyunjun Eun},
  title = {{KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to Diverse Visual Contexts}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure