Abstract
Large Language Models (LLMs) have demonstrated exceptional reasoning capabilities, yet their performance is primarily optimized for high-resource languages like English. For low-resource languages such as Armenian, the lack of large-scale, human-annotated datasets remains a significant barrier. This paper introduces SynDARin, a novel framework that automates the synthesis of high-quality Question Answering (QA) datasets. By leveraging parallel Wikipedia mining, LLM-driven question generation, and rigorous semantic validation, SynDARin produces natural, factually grounded benchmarks. We demonstrate this approach by creating the first Armenian reasoning benchmark, revealing a persistent performance gap between current state-of-the-art models and human benchmarks.
Introduction
The proficiency of LLMs in automated reasoning is largely attributed to the availability of massive datasets like SQuAD. However, for languages with limited digital footprints, collecting such data is economically and logistically challenging. Traditional workarounds, such as direct machine translation of English datasets or purely synthetic generation, often result in "translationese"—unnatural phrasing—or hallucinations. SynDARin addresses these issues by grounding the generation process in human-curated parallel text and employing a strict validation pipeline.
Methodology: The SynDARin Pipeline
The SynDARin framework consists of a three-stage pipeline designed to ensure linguistic naturalness and factual accuracy:
- Parallel Data Mining: Instead of translating English text, the system identifies existing parallel paragraphs in Wikipedia using the Wiki-API and length-matching heuristics. This ensures the source material in the target language is human-written and culturally authentic.
- Synthetic QA Generation: GPT-4 is prompted to generate diverse multiple-choice (MC) questions based on the English versions of the mined paragraphs. This leverages the model's high proficiency in English to create complex reasoning tasks.
- Translation and Validation: The generated questions are translated into the target language. A critical "Validation and Filtering" step follows, using fuzzy substring matching and semantic similarity (multilingual sentence embeddings) to verify that answers are present and contextually relevant within the target language paragraph.
Dataset Characteristics and Quality Control
The primary output of this methodology is a new Armenian QA benchmark comprising 1,235 high-quality samples (987 for training, 247 for testing). The source material spans diverse topics, including history, film, and sports.
A key finding is the necessity of aggressive filtering. The validation pipeline discards approximately 70% of raw generated data. Human evaluation confirmed that this pruning is essential for maintaining dataset integrity, effectively removing hallucinations and translation artifacts. Furthermore, "degeneracy testing"—training models on questions or paragraphs alone—showed performance near random chance (25%), confirming that the dataset requires genuine contextual reasoning rather than exploiting statistical biases.
Experimental Results
We benchmarked several models on the synthesized Armenian dataset to evaluate their reasoning capabilities in a low-resource setting.
As shown in the results, Command-R+ emerged as the leading model, achieving a peak accuracy of 70.9% in a 6-shot configuration. It significantly outperformed GPT-3.5 (59.1%) and smaller fine-tuned models like XLM-RoBERTa, which reached only 39.5%. Despite these results, a substantial gap remains compared to human performance, indicating that reasoning in low-resource contexts remains a frontier challenge for current AI systems.
Conclusion
SynDARin provides a scalable pathway for developing reasoning benchmarks in low-resource languages without the prohibitive costs of manual annotation. By prioritizing human-written source text and rigorous validation, the framework produces high-quality data that exposes the current limitations of LLMs in non-English contexts. Future work will focus on scaling this approach to a broader range of languages and reducing reliance on automated translation tools.
Summary updated