‘Calling on the classical phone’: a distributional model of adjective-noun errors in learners’ English

Aurélie Herbelot, Ekaterina KochmarCOLING 2016

TL;DR

This paper tackles the challenging problem of detecting errors in adjective-noun combinations in English as a Second Language (ESL) learners' writing. They propose a novel approach using "topic coherence" to capture semantic anomalies, achieving state-of-the-art results by integrating context into error detection.

What Problem Does This Paper Address?

Detecting errors in learners' English is a crucial task for improving language learning tools. While grammatical and function word errors have been extensively studied, errors in "content words" – words carrying the main meaning (like adjectives and nouns) – remain a significant challenge. Specifically, this paper focuses on adjective-noun (AN) combinations, where learners often confuse words with similar but distinct meanings (e.g., "classic" vs. "classical"). Existing methods struggle with these semantic errors because they often lack a robust way to model context and semantic relationships. This gap hinders the development of effective automated writing assistance tools for ESL learners.

Methods

The core of the paper's contribution lies in its use of topic coherence to model semantic relationships within AN pairs and their surrounding context.

  • Topic Coherence: This technique, borrowed from topic modeling, measures how semantically related a set of words is. The authors hypothesize that erroneous AN combinations will disrupt the overall semantic coherence of a sentence.
  • Distributional Semantics: They build a distributional semantic space from a large corpus (BNC) where words are represented as vectors based on their co-occurrence patterns. This allows them to calculate word similarities.
  • Contextual Modeling: The system calculates topic coherence for the entire context window around the AN, and also for the context with either the adjective or the noun removed. The difference in coherence scores serves as a feature for error detection.
  • SVM Classifier: A Support Vector Machine (SVM) classifier is trained using these coherence features, along with features derived from an out-of-context (OOC) error detection system (COMPDIST), to predict whether an AN pair is erroneous.
  • Adjective-Dependent Analysis: The authors also investigate whether different types of adjectives require different treatment, exploring class-specific models.

Datasets

  • Cambridge Learner Corpus (CLC): The primary dataset used is an AN dataset extracted from the CLC, specifically focusing on ANs not found in the British National Corpus (BNC). This dataset is annotated with error types (semantic vs. form-related) and suggested corrections, and importantly, includes annotations for both in-context (IC) and out-of-context (OOC) errors. The authors use a subset of 824 contexts from this dataset.
  • British National Corpus (BNC): Used to build the distributional semantic space for calculating word similarities.
  • ukWaC: Used for generating artificial error data.

Key Insights and Ideas

  • Context is King for Semantic Errors: Unlike many other error types, detecting semantic errors in content words heavily relies on understanding the surrounding context. A word might be correct in isolation but wrong in a specific sentence.
  • Topic Coherence as a Semantic Signal: The disruption of semantic coherence is a strong indicator of content word errors. If adding an adjective makes a sentence less semantically "tight," it's likely an error.
  • Adjective Classes Matter: The system's performance varies significantly depending on the adjective. This suggests that a one-size-fits-all approach is suboptimal, and adjective-specific models or features could lead to better results.
  • Artificial Data Limitations: Generating realistic errors for content words automatically is difficult. The subtle semantic nuances of learner errors are hard to replicate, highlighting the continued importance of high-quality, human-annotated learner data.

Key Results

  • The proposed topic coherence-based system achieves 59% accuracy using just three coherence features.
  • Adding adjective-specific features boosts accuracy to 66%, matching previous state-of-the-art results from Kochmar and Briscoe (K&B) when their OOC system was adapted to an in-context setting.
  • Combining the context-sensitive COH system with the context-insensitive COMPDIST system (using feature concatenation) achieved 68% accuracy.
  • A pipeline system combining COMPDIST and COH, with OOC gold annotations, reached 76% accuracy, surpassing human performance estimates. However, using COMPDIST's output instead of gold annotations yielded only a minimal improvement (67%).
  • Analysis of adjective classes showed that sentiment-bearing adjectives (e.g., "funny," "good") were easier to classify, while quantity adjectives (e.g., "large," "small") showed more varied performance.

Limitations and Future Work

  • Data Size Bottleneck: The authors acknowledge that the limited size of the learner dataset makes it difficult to set up robust development sets and thoroughly evaluate adjective-specific models. More data is needed for proper training and evaluation.
  • OOC System Recall: The pipeline system's performance was limited by the low recall of the OOC system, which prioritizes precision to avoid wrongly flagging correct ANs. Improving OOC detection recall is crucial.
  • Artificial Error Generation: While attempted, automatic error generation for content words proved less effective than for function words due to the lack of clear confusion sets and the subtlety of semantic errors.
  • Future Work:
    • Investigating the influence of learners' first language (L1) on semantic errors.
  • Expanding the dataset with more controlled adjective-specific training and testing data.
  • Exploring more sophisticated procedures for automatic error generation, potentially by analyzing large unannotated learner corpora.
  • Further research into adjective-dependent classification and category formation.

Summary updated

BibTeX
@inproceedings{herbelot2016calling,
  author = {Aurélie Herbelot and Ekaterina Kochmar},
  title = {{‘Calling on the classical phone’: a distributional model of adjective-noun errors in learners’ English}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
}
Figure