KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to Diverse Visual Contexts

Taebaek Hwang, Minseo Kim, Gisang Lee, Seonuk Kim, Hyunjun Eun

TL;DR

This paper introduces KRETA, the first large-scale benchmark for Korean text-rich Visual Question Answering (VQA), designed to evaluate both basic text understanding and complex reasoning. KRETA addresses a critical gap for low-resource languages, offering a robust evaluation framework and a semi-automated pipeline that can be adapted for other languages, accelerating multilingual VLM research.

What Problem Does This Paper Address?

Vision-Language Models (VLMs) have made strides in understanding images and text, but their performance on "text-rich" images (like documents, signs, or screenshots) is often limited, especially for languages other than English. Existing VQA benchmarks are scarce for low-resource languages like Korean, hindering the development and evaluation of VLMs for these languages. This lack of resources means VLMs struggle to grasp the nuances of Korean text within diverse visual contexts, impacting their real-world applicability in areas like e-commerce, education, and public information. The authors aim to bridge this gap by creating a comprehensive benchmark specifically for Korean text-rich VQA.

Methods

The core of this work is the KRETA benchmark and its associated semi-automated VQA generation pipeline.

The pipeline involves several key steps:

  1. Stepwise Image Decomposition: Images are first filtered for quality and then processed by multiple VLMs to extract both textual and non-textual information. This structured decomposition aims to capture a richer understanding than simple OCR.
  2. QA Candidates Generation: Using the structured captions, LLMs generate question-answer pairs. This is done using a dual-level reasoning framework:
  3. System 1: Focuses on basic text recognition and understanding.
  4. System 2: Targets advanced reasoning, requiring contextual understanding, multi-step decision-making, and integration of external knowledge.
  5. QA Evaluation and Voting: Multiple VLMs evaluate the generated QA candidates using a set of five metrics for System 1 and seven for System 2, ensuring quality and selecting the best QA pair.
  6. Hard Negatives Generation: LLMs create challenging incorrect answer options that are similar to the correct answer to make the multiple-choice format more robust.
  7. Human Annotation Refinement: Final QA pairs undergo human review for accuracy and alignment.

The benchmark itself is categorized by 15 domains (aligned with the Korean Standard Industrial Classification) and 26 image types, allowing for fine-grained analysis.

Datasets

The paper introduces KRETA, a new benchmark comprising 2,577 samples. This dataset includes:

  • 1,426 System 1 QA pairs (basic recognition and understanding).
  • 1,151 System 2 QA pairs (advanced reasoning).

KRETA is unique in that it exclusively uses native Korean imagery and questions, avoiding reliance on translated English resources. This contrasts with some existing multilingual VQA datasets that might use translated questions or non-Korean images. The authors also mention adapting data from official Korean CSAT (College Scholastic Ability Test) exams.

Key Insights and Ideas

  • System 2 is the Bottleneck: While VLMs perform well on basic text recognition (System 1), there's a significant performance drop for advanced reasoning tasks (System 2), especially for open-source models. This highlights a general deficiency in complex, multi-step reasoning capabilities.
  • Low-Resource Language Challenges: Korean VLMs struggle due to a lack of domain-specific and cultural knowledge, and limited exposure to Korean-contextualized data during training. This necessitates targeted training and fine-tuning.
  • Domain and Image Type Matter: Performance varies drastically across different domains and image types. Models excel in some areas (e.g., closed models in CSAT History) but falter in others (e.g., open-source models on banners and store signs). This emphasizes the need for specialized training and evaluation.
  • Model Size vs. Performance: Generally, larger models show better performance, but there are exceptions, suggesting that scaling alone isn't enough; the quality and relevance of training data (especially for low-resource languages) are crucial.
  • CoT Prompting Benefits: Chain-of-Thought prompting can improve System 2 performance, but only for models with sufficient reasoning capacity. For smaller or less capable models, it can even be detrimental.

Key Results

  • Overall Performance Gap: Closed-source models outperform open-source models by an average of 24.2% overall, with a System 2 gap reaching 44.4%.
  • System 1 vs. System 2: Gemini-2.0-flash achieves 98.0% on System 1, but performance drops significantly for System 2. Open-source models like Qwen2.5-VL-7B drop from 94.5% (System 1) to 36.1% (System 2).
  • Domain-Specific Gaps: The System 2 gap between closed and open-source models can be as high as 40.7 percentage points in CSAT History.
  • Image Type Differences: Book Covers show a large gap (39.0% in System 2), while Street Signs exhibit an unusual pattern where System 2 accuracy (93.1%) exceeds System 1 (87.0%) for closed models.
  • Model Size Correlation: Increasing model parameters generally leads to better performance, but the anomaly in Qwen2.5-VL suggests that scaling needs careful consideration of low-resource language data balance.

Limitations and Future Work

  • Single-Image, Multiple-Choice: KRETA is currently limited to single-image, multiple-choice VQA. Future work could extend to multi-image, video-based scenarios, and open-ended generation tasks.
  • System 2 Granularity: The current System 2 category conflates several reasoning functions. A more fine-grained taxonomy could better expose specific model weaknesses.
  • CoT Optimization: While CoT prompting shows promise, further research into alternative CoT techniques and optimization strategies for both closed and open-source models is needed.
  • Pipeline Transferability: The authors express hope that their VQA generation pipeline can be adapted for other low-resource languages, facilitating the development of culturally and linguistically tailored VLMs.

Summary updated

BibTeX
@inproceedings{hwang2025kreta,
  author = {Taebaek Hwang and Minseo Kim and Gisang Lee and Seonuk Kim and Hyunjun Eun},
  title = {{KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to Diverse Visual Contexts}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure