Automatic Input Enrichment for Selecting Reading Material: An Online Study with English Teachers

Maria Chinkina, Ankita Oswal, Detmar MeurersNAACL 2018

TL;DR

This paper introduces FLAIR, an automatic input enrichment system that re-ranks web search results to prioritize texts containing specific linguistic forms relevant for language learning. An online study with English teachers found that FLAIR-enriched texts were preferred over standard search engine results, suggesting a promising avenue for supporting teachers in finding suitable reading materials.

What Problem Does This Paper Address?

Language acquisition heavily relies on comprehensible input, meaning learners need exposure to language that is both understandable and contains the linguistic forms they are meant to acquire. Teachers face the challenge of finding reading materials that are not only relevant to the topic but also rich in specific grammatical and lexical structures. Manually searching for such materials is time-consuming and inefficient. This paper addresses the need for automated tools to assist teachers in this process by systematically identifying and prioritizing texts that contain target linguistic forms.

Methods

The core of the paper is the FLAIR system, which acts as a "linguistically aware web search engine." FLAIR takes standard web search results and re-ranks them based on the presence and frequency of specified linguistic forms. This is achieved through NLP analysis of the texts. The study then compares FLAIR's output to standard web search results (specifically, Microsoft Bing) in an online experiment. Teachers were presented with pairs of articles on the same topic, one from Bing and one re-ranked by FLAIR, and asked to rate their relevance and linguistic richness, and then choose which they would assign to their students.

Datasets

The study used a collection of news articles from Reuters. For the experiment, 60 news articles were selected, covering ten popular topics. These articles were paired, with one from a standard Bing search and the other from FLAIR's re-ranked results. The linguistic forms analyzed were categorized into three types based on their frequency in a 2400-article corpus: frequent (regular/irregular verbs), mixed (present simple/continuous), and infrequent (comparative degree of short/long adjectives and adverbs).

Key Insights and Ideas

  • Teacher Preference for Enriched Input: Teachers demonstrably preferred texts enriched by FLAIR over those from a standard search engine, indicating a practical need for such tools.
  • Linguistic Richness vs. Topic Relevance Trade-off (or lack thereof): Contrary to expectations, FLAIR-enriched texts were not significantly less relevant to the topic than standard search results. This suggests that texts not appearing in top search results can still be highly relevant and pedagogically valuable.
  • Benefit for Infrequent Forms: While not statistically significant, there was a trend suggesting that FLAIR is particularly beneficial for identifying texts with less frequent linguistic forms, which are harder to find through standard searches.
  • Beyond Top Search Results: The study highlights that valuable learning materials might be buried deep within search results, and re-ranking systems like FLAIR can surface them effectively.

Key Results

  • Preference: 71% of participants chose FLAIR-enriched texts over Bing results when selecting reading material.
  • Linguistic Representation: FLAIR-ranked texts received significantly higher ratings for linguistic form representation (M=3.22) compared to Bing results (M=2.51).
  • Topic Relevance: Texts provided by FLAIR were perceived as slightly more relevant to the topic (M=3.67) than those from Bing (M=3.58), though this difference was not statistically significant. Equivalence tests confirmed comparable content relevance.
  • Form Frequency Impact: While not statistically significant, the preference for FLAIR was highest for the "mixed" category of linguistic forms, followed by "infrequent," and then "frequent." This suggests that the system's re-ranking is most impactful when forms are less common.

Limitations and Future Work

The authors acknowledge that the study did not directly compare automatic input enrichment to web search as an end in itself, but rather focused on the impact of FLAIR's re-ranking. A limitation is that the study did not test the actual learning outcomes of students using FLAIR-enriched materials, only teacher preference. Future work could involve end-to-end empirical studies to assess learning outcomes. Additionally, the authors suggest exploring how FLAIR performs with more specific queries or less represented languages, and investigating the integration of components for automatic exercise generation and sense disambiguation to further enhance the system's utility.

Summary updated

BibTeX
@inproceedings{chinkina2018automatic,
  author = {Maria Chinkina and Ankita Oswal and Detmar Meurers},
  title = {{Automatic Input Enrichment for Selecting Reading Material: An Online Study with English Teachers}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure