Assessing the Helpfulness of Learning Materials with Inference-Based Learner-Like Agent

Yun-Hsuan Jen, Chieh-Yang Huang, Mei-Hua Chen, Ting-Hao K. Huang, Lun-Wei Ku

Distinguishing between near-synonyms—words like small vs. little or refuse vs. reject—is a major hurdle for English-as-a-second-language (ESL) learners. While learners often turn to example sentences to understand these nuances, not all examples are created equal. Some clarify the difference, while others are ambiguous or even confusing. Existing automated systems for recommending example sentences usually rely on "hand-crafted" scores like sentence length or word frequency. However, these rules are brittle; a rule that works for sentiment-based differences might fail for grammatical ones. This research introduces a way to measure "helpfulness" that adapts to the specific nuances of any given word pair by simulating the human learning process.

Methodology: The Learner-Like Agent

The authors propose the Inference-Based Learner-Like Agent (EMLA). The core philosophy is operational: if a learning material is good, a learner should perform better on a test after seeing it. EMLA simulates this by treating the evaluation of educational material as a performance-based feedback loop.

Conceptual framework of the Learner-Like Agent compared to a human learner.

Key technical components include:

  • Entailment Modeling: Instead of just looking at the context of a word, EMLA treats the problem as a Natural Language Inference (NLI) task. It treats the example sentence as a "premise" and the test question as a "hypothesis." If the example sentence entails the correct answer in the question, the agent identifies it as helpful.
  • Perturbed Instances: To ensure the agent actually "learns" from the provided material rather than just relying on its pre-trained knowledge (like BERT's internal weights), the researchers introduced "trick" training data. They swapped target words with their near-synonyms in some examples. This forces the model to pay attention to the specific evidence provided in the example sentence to reach the right conclusion.
  • Comparison Model (CMLA): The researchers compared EMLA against a Context Modeling agent (CMLA), which uses standard BERT-based fill-in-the-blank (FITB) logic without the inference-based entailment layer.

Experimental Setup

The system was evaluated using several datasets:

  • Near-Synonym Pairs: 30 pairs of commonly confused English words (e.g., job/task, briefly/shortly) selected by ESL experts.
  • Expert Evaluation Set: A dataset of 600 example sentences where an ESL expert labeled which sentences were truly "helpful" for clarifying nuances.
  • User Study: Performance data from 29 Chinese-speaking college freshmen majoring in English to validate the agent's findings against real human learning outcomes.

Key Results and Insights

EMLA demonstrated a superior ability to identify high-quality learning materials. In sentence selection tasks, EMLA significantly outperformed the baseline Context Modeling (CMLA) and a Gaussian Mixture Model (GMM), achieving an F1 score of 0.45 (compared to 0.38 and 0.35 for the baselines).

Lexical choice accuracy across thirty near-synonym wordpairs for EMLA and baseline models.

Key takeaways from the performance data include:

  • Inference > Context: Standard language models are excellent at predicting words based on context, but they don't necessarily "reason" about why one word fits better than a near-synonym. By framing the task as entailment, EMLA is forced to find the logical link between the example and the question.
  • The Necessity of Perturbation: Without the perturbed (swapped) instances during training, models tended to ignore the "learning material" and guess based on prior training. Perturbation acts as a regularizer that anchors the agent's behavior to the provided evidence.
  • Human Impact: In a classroom study, students who studied with EMLA-selected sentences showed the most improvement. Specifically, EMLA improved the scores of 17% more students compared to other models.

Conclusion

The "Learner-Like Agent" concept suggests that "helpfulness" does not need a complex linguistic definition; it can be defined by its impact on task performance. While this study focused on single example sentences for 30 word pairs, the framework could potentially scale to evaluate more complex educational content, such as longer texts or videos, acting as an automated "AI tutor" that vets curriculum quality before it reaches human students.

Summary updated

Abstract

Many English-as-a-second language learners have trouble using near-synonym words (e.g., small vs.little; briefly vs.shortly) correctly, and often look for example sentences to learn how two nearly synonymous terms differ. Prior work uses hand-crafted scores to recommend sentences but has difficulty in adopting such scores to all the near-synonyms as near-synonyms differ in various ways. We notice that the helpfulness of the learning material would reflect on the learners' performance. Thus, we propose the inference-based learner-like agent to mimic learner behavior and identify good learning materials by examining the agent's performance. To enable the agent to behave like a learner, we leverage entailment modeling's capability of inferring answers from the provided materials. Experimental results show that the proposed agent is equipped with good learner-like behavior to achieve the best performance in both fill-in-the-blank (FITB) and good example sentence selection tasks. We further conduct a classroom user study with college ESL learners. The results of the user study show that the proposed agent can find out example sentences that help students learn more easily and efficiently. Compared to other models, the proposed agent improves the score of more than 17% of students after learning.
BibTeX
@inproceedings{jen2020assessing,
  author = {Yun-Hsuan Jen and Chieh-Yang Huang and Mei-Hua Chen and Ting-Hao K. Huang and Lun-Wei Ku},
  title = {{Assessing the Helpfulness of Learning Materials with Inference-Based Learner-Like Agent}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.312},
}
Figure