Improving Automatic Evaluation of Acceptability Based on Language Models with a Coarse Sentence Representation

Vijay Daultani, Naoaki OkazakiPACLIC 2022 · OpenReview

Introduction

Determining whether a sentence is "acceptable"—meaning fluent and grammatically sound—is a critical task in Natural Language Processing (NLP). While Language Models (LMs) are commonly used to score acceptability, they suffer from a significant bias: they are highly sensitive to word frequency. A sentence containing rare names or Out-of-Vocabulary (OOV) words often receives a lower score than a structurally identical sentence with common words, regardless of its actual grammatical correctness.

The Problem: Frequency Bias in Acceptability Metrics

Standard metrics like the Syntactic Log Odds Ratio (SLOR) attempt to normalize for word frequency, but they often fail to fully eliminate the "rare word penalty." When an LM encounters a rare entity, such as a specific geographic location or a complex organization name, the resulting low probability score can distort the overall assessment of the sentence's structure. The challenge lies in evaluating the structure of a sentence independently of the specific lexical choices for named entities.

Proposed Method: Replace Named Entity (RNE)

To address this, the researchers propose a transformation strategy called Replace Named Entity (RNE). This method creates a "coarse representation" of a sentence by stripping away specific names and replacing them with general category labels.

The process involves two primary steps:

  1. Named Entity Identification: Utilizing the spaCy library to identify and classify proper nouns into 18 distinct types (e.g., PERSON, ORG, DATE).
  2. Replacement: Substituting each identified entity with its corresponding label.
Example of the RNE transformation process, converting specific entities into general category labels.

By training n-gram models on this transformed data, the model learns the structural transitions between parts of speech and entity types, effectively ignoring the noise introduced by rare vocabulary.

Experimental Setup

The models were trained on the British National Corpus (BNC), which contains approximately 6.07 million sentences. Evaluation was conducted using the SMOG (Statistical Model of Grammaticality) corpus, specifically focusing on datasets with human acceptability ratings:

  • BNC: 5,250 rated sentences.
  • ENWIKI: 2,500 sentences from Wikipedia.
  • ADGER: 300 sentences from linguistics literature.
  • ADGER-FILTERED: A 133-sentence subset focused on syntactic violations.

Key Results

The RNE method demonstrated substantial improvements in the correlation between automated scores and human judgments. As shown in the results for the BNC dataset, the application of RNE (SP+RNE) consistently outperformed the standard baseline (SP) across various measures and n-gram sizes.

Pearson correlation coefficients for different measures on the BNC dataset, comparing standard processing (SP) with the Replace Named Entity (RNE) method.

Notably, for the 4-gram model using the LogProb metric, the correlation jumped from 0.42 to 0.65. The most significant gains were observed in the Norm LP (Div) and SLOR metrics, where correlations reached as high as 0.73. These results suggest that focusing on coarse representations allows the model to better capture the underlying syntactic patterns that humans use to judge acceptability.

Conclusion

The RNE strategy effectively mitigates the frequency bias inherent in standard language models. By abstracting named entities into general categories, the model prioritizes sentence structure over vocabulary rarity. Future work may explore extending this coarse representation to other parts of speech or developing dynamic replacement strategies to further refine automatic acceptability evaluation.

Summary updated

BibTeX
@inproceedings{daultani2022improving,
  author = {Vijay Daultani and Naoaki Okazaki},
  title = {{Improving Automatic Evaluation of Acceptability Based on Language Models with a Coarse Sentence Representation}},
  booktitle = {PACLIC},
  year = {2022},
  url = {https://openreview.net/forum?id=3wm1Nf-NarY},
}
Figure