TL;DR
This paper introduces the Trojsten Benchmark, a new dataset of 1,108 Slovak STEM competition problems, to evaluate LLM problem-solving in a low-resource language. They developed an LLM-based grading framework using GPT-4 and found that while LLMs show promise, performance degrades on more difficult problems and is sensitive to language, even outperforming English translations for GPT-4.
What Problem Does This Paper Address?
Evaluating the problem-solving capabilities of Large Language Models (LLMs) in STEM fields is crucial, but current benchmarks are heavily skewed towards high-resource languages like English. This leaves a significant gap in understanding how LLMs perform on complex, multi-step reasoning tasks in languages with less digital representation. Furthermore, existing evaluation methods often focus on simpler answers or rely on pre-defined rubrics, failing to capture the nuances of open-ended, rubric-gradable STEM problems. This paper aims to address this by creating a benchmark and evaluation framework for Slovak, a low-resource language, to assess LLM performance on challenging competition problems.
Methods
The core of this work involves two main components: a new dataset and an LLM-based grading framework.
- Trojsten Benchmark Dataset: The authors curated 1,108 high-school competition problems from mathematics, physics, and programming in Slovak. These problems were chosen because their PDF formatting makes them difficult to process automatically, suggesting they are less likely to have been included in LLM training data, thus providing a more unbiased evaluation. The dataset includes reference solutions.
- LLM-based Grading Framework: They developed a rubric-based grading methodology. This involves using GPT-4 to first generate a detailed grading rubric for a given problem and its reference solution. Subsequently, GPT-4 is used again to evaluate a student's solution against this generated rubric, assigning a score out of 10. This approach aims to capture the complexity of open-ended responses and provide more nuanced feedback than simple answer checking.
Datasets
- Trojsten Benchmark: A newly created dataset of 1,108 Slovak high-school competition problems in mathematics (361), physics (479), and programming (268). This dataset is designed for low-resource language evaluation and focuses on open-ended, rubric-gradable problems.
- Existing Datasets (for context): The paper references several English-centric datasets like MATH, GSM8K, MGSM, Omni-MATH, CHAMP, and MathOdyssey to highlight the scarcity of low-resource language benchmarks.
Key Insights and Ideas
- Language Sensitivity is Key: LLM performance is not just about translation; the linguistic framing and cultural context of a problem can significantly impact reasoning. GPT-4's performance dropped dramatically when Slovak problems were translated to English, indicating a challenge beyond simple language conversion.
- Difficulty Matters: Similar to human performance, LLMs struggle with more difficult problems. Their scores consistently decrease as problem difficulty increases, suggesting that current LLMs are not yet adept at handling the most complex reasoning tasks.
- LLMs as Graders are Promising, but Imperfect: While GPT-4 can generate rubrics and grade solutions with reasonable agreement with human evaluators (average absolute difference of 1.05 points when following a rubric), it tends to overscore and can sometimes hallucinate or overlook valid alternative approaches.
- Slovak is a Real Challenge: The performance of LLMs on the original Slovak problems was often better than on English translations, underscoring the importance of evaluating models in their target languages, especially for low-resource settings.
Key Results
- LLM Grading Accuracy: GPT-4, when following a rubric, showed an average absolute difference of 1.05 points compared to human evaluators. However, it tended to overestimate scores by an average of 0.9 points.
- Rubric Agreement: Human competition organizers agreed with GPT-4 generated rubrics with a mean Likert score of 3.98 out of 5.
- Model Performance: GPT-40 achieved the highest average score (6.07 out of 10) on the Trojsten Benchmark among the tested models.
- Language Impact: For GPT-4, performance on original Slovak problems (average score 3.30) was significantly higher than on English translations (average score 1.32).
- Model Output Language: LLMs often defaulted to English output even when problems were in Slovak. When they did output in Slovak, the average score was 3.53, compared to 4.14 for English outputs.
Limitations and Future Work
- Rubric Hallucinations: GPT-4 can sometimes hallucinate mathematical relationships or fail to award points for alternative correct solutions, indicating a need for improved rubric generation prompts or external verification tools.
- Error Propagation: The current evaluation method doesn't fully account for error propagation across the rubric generation and grading stages.
- LLM Biases: Biases in LLM training data might affect performance on language-specific nuances.
- Human Oversight: The authors strongly advise that LLM grading systems should only be used as a hint for human evaluators, not as a replacement, due to their current unreliability. Future work could focus on mitigating these limitations, exploring more robust prompting techniques, and further investigating language-specific reasoning challenges.
Summary updated