TL;DR
This paper tackles the challenge of automatically generating distractors for English fill-in-the-blank exercises, a crucial component of university entrance exams. By categorizing questions into grammar, function word, and context types, the authors propose a novel method that tailors distractor generation to each type, leading to more effective and realistic distractors that better reflect examinee performance.
What Problem Does This Paper Address?
Creating effective distractors for fill-in-the-blank (cloze) tests in English proficiency exams is a labor-intensive process requiring expert knowledge. While previous automated methods have focused on generating semantically similar words, they often apply a one-size-fits-all approach. This fails to account for the diverse nature of actual entrance exam questions, which are designed to test different linguistic skills (grammar, vocabulary, idiomatic usage). This lack of specialization leads to distractors that may not accurately reflect the difficulty or intent of the original question, potentially biasing evaluation results.
Methods
The core of this paper's contribution lies in its type-specific distractor generation. The authors first manually classify English fill-in-the-blank questions into three categories:
- Grammar: Questions focusing on correct verb conjugations or word forms.
- Function Word: Questions where choices are limited to prepositions, conjunctions, or other grammatical function words.
- Context: Questions testing understanding of collocations, idioms, or semantic appropriateness within a sentence.
For each type, they propose tailored methods, primarily focusing on candidate filtering:
- Grammar: They employ POS (Part-of-Speech) tagging to ensure distractors are grammatically plausible and use a dynamic BERT-based filtering approach to exclude candidates with overly high probabilities, preventing them from being too close to the correct answer.
- Function Word: They leverage a predefined list of function words used in Japanese university entrance exams to ensure distractors are drawn from the correct grammatical category.
- Context: Similar to grammar questions, they use BERT-based filtering to remove candidates that are too likely to be correct, while also aiming for words that co-occur with surrounding words in the carrier sentence.
The overall process involves candidate generation (using methods like fastText embeddings and round-trip machine translation), reranking (using BERT), and then the proposed type-specific filtering.
Datasets
The study utilizes a dataset of 500 randomly selected English fill-in-the-blank questions from Japanese university entrance examinations spanning the years 2017-2021. This dataset was manually annotated by an expert to categorize questions into the three defined types. An additional validation dataset of 500 questions was also used for tuning the methods.
Key Insights and Ideas
- Question Type Matters: The most significant insight is that a generic approach to distractor generation is insufficient for real-world language proficiency tests. Tailoring generation to question type is crucial for creating effective distractors.
- Dynamic BERT Filtering: The proposed dynamic BERT filtering, which adjusts thresholds based on the correct word's probability, is more effective than static thresholding for identifying and excluding overly probable candidates.
- Leveraging Domain-Specific Resources: The use of a specific list of function words for Japanese university entrance exams highlights the importance of incorporating domain-specific knowledge for better distractor generation.
- Improved Correlation with Examinee Performance: The type-specific method shows a stronger correlation with actual examinee performance, suggesting it generates distractors that better mimic the difficulty and nature of real exam questions.
Key Results
- Automatic Evaluation: The proposed method achieved the best F1-score performance in 9 out of 12 settings across the three question types, demonstrating its overall superiority. The improvement was particularly notable for function word questions.
- Manual Evaluation: The proposed method showed the highest correlation (Pearson, Spearman, and Kendall) with the performance of actual entrance examinations, indicating its effectiveness in creating distractors that align with how students perform on real tests.
- Qualitative Improvement: Examples show that the proposed method generates more appropriate distractors, such as correct conjugated forms for grammar questions and relevant function words for function word questions, unlike generic methods.
Limitations and Future Work
- Limitations: The paper acknowledges that the current method's performance on context questions is not significantly different from existing methods until the top five candidates, suggesting room for improvement in this category. The corpus size for training and evaluation, while substantial, could be expanded.
- Future Work: The authors plan to expand the corpus size by automatically estimating question types. They also aim to explore supervised learning approaches for generating distractors, which could potentially lead to even more sophisticated and accurate results.
Summary updated