TL;DR
This paper tackles the challenge of "Multiple Admissibility" (MA) in language learning systems, where multiple grammatically correct forms of a word can fit a given context. The authors propose a supervised grammaticality judgment task to train a neural network to identify these cases, using artificially generated errors and real user data to build their models and test sets. This work is crucial for improving the feedback provided by intelligent tutoring systems, preventing learners from receiving incorrect negative feedback.
What Problem Does This Paper Address?
Intelligent tutoring systems (ITS) and computer-aided language learning (CALL) platforms, like the Revita system discussed in this paper, often use exercises like "fill-in-the-blank" to help learners practice grammar. In these exercises, a word is removed from a text, and the learner is given a hint (usually the base form or lemma of the word) to produce the correct grammatical form. A significant problem arises when more than one grammatical form of a word is valid in the given context – this is termed "Multiple Admissibility" (MA). If the system only accepts one specific form and rejects others that are also grammatically and semantically correct, it provides negative feedback to the learner, which can be discouraging and misleading. This paper aims to address this by developing a system that can recognize these instances of MA, thereby improving the accuracy and helpfulness of language learning feedback.
Methods
The authors frame the problem of identifying MA as a supervised grammaticality judgment task. They train a neural network, specifically a bidirectional LSTM followed by a Multi-layer Perceptron (MLP), to classify sentences as grammatical or ungrammatical. A key aspect of their approach is the generation of artificial grammatical errors to create a training dataset. This involves taking sentences from a large Russian corpus, sliding a window across them, and then replacing a target token with a random grammatical form from its paradigm, provided the token is unambiguous and meets a frequency threshold. This process generates "ungrammatical" instances. The training dataset is balanced with both artificially generated errors and correct sentences. For evaluation, they create several test sets, including one specifically designed to contain MA instances, and others with real user errors.
Datasets
- Training/Validation Data: Generated artificially by injecting errors into sentences from the open-source "Taiga" Russian corpus (809M words, including news and literary segments).
- Test Data:
- Test Set A: Sentences exhibiting MA, manually annotated.
- Test Set B (Random Correct): 500 randomly sampled correct sentences from a corpus not used for training/validation and not present in the Revita database.
- Test Set C (Correct & Incorrect): A mix of correct sentences from Revita's database and sentences with grammatical errors made by Revita users.
- Test Set D (Pragmatic Errors): Sentences that are grammatically correct but pragmatically unnatural.
- Real User Data: 2884 answers from the Revita database automatically marked as "incorrect," manually annotated into categories like Grammatical error, Non-word error, Multiple-choice, Multi-admissible, Pragmatic error, and Broken.
- Advanced Student Data: A subset of the real user data from students with C1 CEFR proficiency.
Key Insights and Ideas
- MA is More Prevalent for Advanced Learners: The study found that for advanced learners (C1 CEFR level), a significantly higher percentage (15.8%) of their "incorrect" answers were actually grammatically and semantically valid in context, highlighting the increased relevance of MA for more proficient users.
- Diverse Sources of MA: The paper provides a detailed classification of MA contexts in Russian, including tense/aspect, number agreement, adjective forms, case usage, and prepositions, demonstrating the complexity of the problem.
- Artificial Data Generation is Crucial: Due to the scarcity of naturally occurring, annotated MA data, the authors' method of simulating grammatical errors is essential for training robust models.
- Model Struggles with Syntactic Complexity: While the model performs well on general grammaticality judgment, it shows lower accuracy on MA contexts compared to some random correct sentences, suggesting that distinguishing between multiple valid forms is harder than identifying outright errors.
- Pragmatic Errors Pose a Challenge: Sentences that are grammatically correct but pragmatically unnatural present the greatest difficulty for the model, as it was not explicitly trained to detect them.
Key Results
- The trained neural network achieved an overall accuracy of 85.9% on the test set designed to evaluate MA.
- The model performed particularly well on MA contexts related to Perfective/Imperfective aspect and Gerund/Other verb forms (92.8% accuracy) and Case (91.7% accuracy).
- Accuracy was lower for Number agreement (81.0%) and Short/Full adjectives (77.7%).
- On the "Random Correct" test set (B), the model achieved 92.3% accuracy, indicating better performance on standard correct sentences than on MA instances.
- The authors' approach to generating artificial errors resulted in a training dataset where 3% of the generated instances were identified as "Multi-admissible" by annotators.
Limitations and Future Work
- Limited Scope of MA: The current work focuses on MA of multiple surface forms of the same lemma, excluding synonyms that could also fit the context.
- Pragmatic Error Detection: The model was not explicitly trained to detect pragmatic errors, which proved to be the most challenging type of error.
- Future Work:
- Extend the training data by marking numbers with special tokens or mapping them to words, and marking rare words with their part of speech.
- Incorporate syntactic information (parsing) into the model, potentially through a multitask learning approach.
- Explore the use of larger, state-of-the-art language models.
- Conduct additional annotation of student data to expand the dataset.
- Extend experiments to other languages with their specific MA types.
- Investigate the problem of MA in free-form answers, such as essays.
Summary updated