PersianMCQ-Instruct: A Comprehensive Resource for Generating Multiple-Choice Questions in Persian

Kamyar Zeinalipour, Neda Jamshidi, Fahimeh Akbari, Marco Maggini, Monica Bianchini, Marco GoriWorkshops @ COLING 2025 · OpenReview

Abstract

We present PersianMCQ-Instruct, a comprehensive resource that includes a dataset and advanced models for generating multiple-choice questions (MCQs) in standard Iranian Persian, a low-resource language spoken by over 80 million people. This resource features three state-of-the-art models for Persian MCQ generation: PMCQ-Gemma2-9b, PMCQ-Llama3.1-8b, and PMCQ-Mistral-7B. Inspired by the Agent Instruct framework and GPT-4o, we created the dataset by curating over 4,000 unique Persian Wikipedia pages, resulting in three MCQs per page and a total of over 12,000 questions. To ensure the quality of this dataset, we conducted human evaluations and model fine-tuning, both of which demonstrated significant performance improvements in Persian MCQ generation. The dataset and models are publicly available, offering valuable tools for researchers and educators, with particular benefits for advancing Persian-language educational technology.

Introduction

While Large Language Models (LLMs) have demonstrated remarkable proficiency in English-centric educational tasks, Persian remains a "low-resource" language in the field of Natural Language Processing (NLP). Educators and developers in Persian-speaking regions have long lacked high-quality, automated tools to generate assessments. Existing datasets are often limited in size or are mere translations of English benchmarks, which frequently fail to capture the unique linguistic nuances and cultural context of Farsi.

The challenge in automated assessment is not merely generating a question, but producing high-quality items with plausible "distractors" (incorrect answers) that meet rigorous educational standards. PersianMCQ-Instruct addresses this gap by providing both a specialized dataset and optimized models tailored for Persian MCQ generation.

Methodology

The researchers employed a sophisticated three-step pipeline inspired by the Agent Instruct framework to build the PersianMCQ-Instruct dataset. This workflow ensures that the generated content is not only factually accurate but also pedagogically sound.

The PersianMCQ-Instruct pipeline, illustrating the flow from data collection to model training.

The Three-Step Generation Pipeline

  1. Text Augmentation: Raw Persian Wikipedia articles are transformed into "argumentative passages" using GPT-4o. This step highlights key concepts and reformats the text to be better suited for question generation.
  2. MCQ Generation: The model generates three MCQs per passage, ensuring each item includes one correct answer and three distractors.
  3. MCQ Refinement: A final pass increases the complexity of the questions and the subtlety of the distractors. This ensures that incorrect options are plausible rather than "obviously wrong," thereby effectively challenging the learner's comprehension.
The three primary prompts used for text augmentation, MCQ generation, and MCQ refinement.

Model Fine-Tuning

Following data collection, the researchers utilized LoRA (Low-Rank Adaptation) to fine-tune three open-weight models: Gemma2-9b, Llama3.1-8b, and Mistral-7b. This approach allowed for the injection of specialized MCQ-generation capabilities into these models without the prohibitive computational costs associated with full parameter fine-tuning.

The PersianMCQ-Instruct Dataset

The primary contribution of this work is the PersianMCQ-Instruct dataset, which consists of over 12,000 high-quality MCQ pairs derived from 4,159 unique Persian Wikipedia pages. To ensure quality, the dataset was validated through a five-point human rating system (Rating A to E). The results were impressive, with 86.8% of the questions in the dataset receiving the highest quality rating (Rating A).

Key Results and Insights

The impact of fine-tuning on open-source models was dramatic. Before fine-tuning, base models like Llama 3.1-8b-Instruct produced 0% "Rating A" questions in the authors' evaluation. After fine-tuning on the PersianMCQ-Instruct dataset, this figure jumped to 81.5%.

  • PMCQ-Llama3.1-8b emerged as the top performer, achieving an overall human rating of 4.51/5.0, a significant improvement over the base model's 3.31.
  • PMCQ-Mistral-7b showed the most substantial leap, moving from a nearly unusable 1.30 rating to a respectable 4.32.
  • Quantitative Metrics: Fine-tuned models showed consistent improvements in BERTScore (F1 scores exceeding 0.90), indicating high alignment with the source context.

These results demonstrate that high-end "teacher" models like GPT-4o can be used to generate synthetic datasets that effectively "teach" smaller, open-source models specialized tasks in low-resource languages.

Limitations and Future Work

Despite these advancements, several challenges remain. The models currently rely exclusively on Wikipedia data, which may limit their performance on highly specialized topics like advanced mathematics or creative, non-factual content. Additionally, while the models excel at fact-based questions, they still face hurdles with complex inference-based questions requiring deep logical synthesis.

Future work will focus on expanding the dataset to include more diverse subjects and involving a broader pool of native Persian speakers to further refine the models' grasp of subtle linguistic nuances and complex reasoning.

Summary updated

Abstract

We present PersianMCQ-Instruct, a comprehensive resource that includes a dataset and advanced models for generating multiple-choice questions (MCQs) in standard Iranian Persian, a low-resource language spoken by over 80 million people. This resource features three state-of-the-art models for Persian MCQ generation: PMCQ-Gemma2-9b, PMCQ-Llama3.1-8b, and PMCQ-Mistral-7B. Inspired by the Agent Instruct framework and GPT-4o, we created the dataset by curating over 4,000 unique Persian Wikipedia pages, resulting in three MCQs per page and a total of over 12,000 questions. To ensure the quality of this dataset, we conducted human evaluations and model fine-tuning, both of which demonstrated significant performance improvements in Persian MCQ generation. The dataset and models are publicly available, offering valuable tools for researchers and educators, with particular benefits for advancing Persian-language educational technology.
BibTeX
@inproceedings{zeinalipour2025persianmcq,
  author = {Kamyar Zeinalipour and Neda Jamshidi and Fahimeh Akbari and Marco Maggini and Monica Bianchini and Marco Gori},
  title = {{PersianMCQ-Instruct: A Comprehensive Resource for Generating Multiple-Choice Questions in Persian}},
  booktitle = {Workshops @ COLING},
  year = {2025},
  url = {https://openreview.net/forum?id=dlfc0CnC7I},
}
Figure