Ko-MuSR: A Multistep Soft Reasoning Benchmark for LLMs Capable of Understanding Korean

Chanwoo Park, Suyoung Park, JiA Kang, Jongyeon Park, Sangho Kim, Hyunji M. Park, Sumin Bae, Mingyu Kang, Jaejin LeearXiv 2025 · DOI · arXiv

Abstract

This paper introduces Ko-MuSR, the first benchmark specifically designed to evaluate multi-step "soft" reasoning in long-context Korean narratives. While Large Language Models (LLMs) have demonstrated proficiency in Korean, existing benchmarks often focus on short-form text or suffer from data contamination. Ko-MuSR addresses these issues by utilizing a three-stage synthetic generation pipeline to create complex, culturally relevant stories. Our evaluation of various models reveals that reasoning capabilities are largely language-agnostic, with massive multilingual models often outperforming Korean-specialized counterparts. This suggests that structural logic transfers effectively across languages when models are trained on sufficiently large datasets.

Introduction

The evaluation of deep reasoning in Large Language Models (LLMs) is a critical frontier in natural language processing. In the Korean language domain, most existing benchmarks are limited to short snippets, coding, or mathematics. Long-context benchmarks like CSAT-QA (based on the Korean "Suneung" exam) are frequently used but are prone to data contamination, as these public exams likely appear in the training corpora of modern LLMs. Ko-MuSR is developed to provide a clean, uncontaminated environment for testing multi-step reasoning through long, complex narratives. It aims to determine whether reasoning is a universal skill or one that requires deep linguistic specialization.

Methodology

The construction of Ko-MuSR follows a rigorous three-stage pipeline adapted from the English MuSR framework. This process ensures that the resulting narratives are logically sound while maintaining natural language quality.

  1. Sampling: Story backgrounds, including character names, locations, and tools, are sampled and tailored to Korean cultural contexts.
  2. Reasoning Tree Expansion: High-end models (e.g., GPT-4o and o1) are used to build a logical "tree" of facts and deductions.
  3. Narrative Generation: These logic trees are transformed into natural-sounding Korean stories, typically between 500 and 1,000 words.
The three-stage construction process for Ko-MuSR: Sampling, Reasoning Tree Expansion, and Narrative Generation.

To ensure the benchmark's integrity, a human-in-the-loop verification process was employed. Human annotators solved every instance to confirm they were logically sound and answerable solely from the text. Instances without consensus were discarded.

The Ko-MuSR Dataset

The dataset consists of 750 total instances, with 250 instances for each of the three subtasks:

  • Murder Mysteries: Identifying a culprit based on "means, motive, and opportunity."
  • Object Placements: Tracking object locations as characters move them, requiring an understanding of characters' beliefs (Theory of Mind).
  • Team Allocations: Assigning individuals to roles based on unique skills and interpersonal dynamics.

Experimental Results and Insights

The study evaluated a wide range of models, from Small Language Models (SLMs) to massive multilingual giants.

Performance comparison of Qwen3 and Exaone models across different scales and prompting strategies on Ko-MuSR tasks.

A primary finding is that reasoning is largely language-agnostic. Multilingual models trained on massive datasets (30+ trillion tokens) significantly outperformed Korean-specialized models. For instance, Qwen3 32B emerged as a top performer, achieving 89.0% accuracy on the Korean Team Allocation task and 76.0% on Korean Murder Mysteries when using 3-shot Chain-of-Thought (CoT) prompting. This suggests that structural logic learned in one language (like English) carries over to Korean, provided the model has a basic grasp of the language.

Scale proved more significant than linguistic specialization. Models like Qwen3 32B outperformed specialized models like HyperCLOVA X. Furthermore, prompting techniques like CoT and task-specific hints acted as force multipliers for larger models, bringing them closer to human performance levels (85.5% - 93.8%). However, SLMs showed inconsistent results, with complex prompts sometimes decreasing accuracy, suggesting they may be overwhelmed by the additional context.

Conclusion

Ko-MuSR provides a robust framework for evaluating multi-step reasoning in Korean. The results underscore the importance of model scale and the cross-lingual nature of reasoning. While LLMs are nearing human-level performance in some tasks, they still struggle with complex mental state tracking in "Object Placement." Future work will involve expanding the benchmark's reasoning types and investigating better distillation techniques for smaller, language-specific models.

Summary updated

Abstract

We present Ko-MuSR, the first benchmark to comprehensively evaluate multistep, soft reasoning in long Korean narratives while minimizing data contamination. Built following MuSR, Ko-MuSR features fully Korean narratives, reasoning chains, and multiple-choice questions verified by human annotators for logical consistency and answerability. Evaluations of four large language models -- two multilingual and two Korean-specialized -- show that multilingual models outperform Korean-focused ones even in Korean reasoning tasks, indicating cross-lingual generalization of reasoning ability. Carefully designed prompting strategies, which combine few-shot examples, reasoning traces, and task-specific hints, further boost accuracy, approaching human-level performance. Ko-MuSR offers a solid foundation for advancing Korean NLP by enabling systematic evaluation of long-context reasoning and prompting strategies.
BibTeX
@misc{park2025ko,
  author = {Chanwoo Park and Suyoung Park and JiA Kang and Jongyeon Park and Sangho Kim and Hyunji M. Park and Sumin Bae and Mingyu Kang and Jaejin Lee},
  title = {{Ko-MuSR: A Multistep Soft Reasoning Benchmark for LLMs Capable of Understanding Korean}},
  year = {2025},
  eprint = {2510.24150},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.24150},
}
Figure