ERAS: Evaluating the Robustness of Chinese NLP Models to Morphological Garden Path Errors

Qinchan Li, Sophie HaoNAACL 2025 · DOI

Introduction: The Challenge of Chinese Word Segmentation

Unlike English, where spaces provide explicit word boundaries, Chinese text is a continuous string of characters. To process this text, NLP models must perform word segmentation—either as an explicit preprocessing step or implicitly within a neural architecture. This lack of clear boundaries gives rise to "morphological garden paths": sequences of characters that are locally ambiguous but globally clear.

A garden path occurs when a model is lured into an incorrect segmentation by a high-frequency character sequence (a "canary word") that doesn't actually form a word in the specific sentence context. For example, the trigram 留心机 could be parsed as 留 (with) + 心机 (caution) or incorrectly as 留心 (take note of) + 机 (machine). While the latter is a valid word pair, it renders the rest of the sentence ungrammatical.

An example of an ungrammatical segmentation resulting from a morphological garden path.

Researchers from NYU have introduced ERAS (Evaluation of Robustness to Locally Ambiguous Segmentation), a new benchmark designed to reveal how state-of-the-art models handle these ambiguities.

The ERAS Benchmark

The ERAS benchmark utilizes a "minimal pair" approach to isolate the impact of garden path errors. Each entry consists of:

  • Test Sentence: Contains a three-character "test site" that is locally ambiguous.
  • Control Sentence: A nearly identical sentence where the ambiguity is removed, typically by replacing one word with a synonym that does not overlap with adjacent characters.
The minimal pair approach: comparing a locally ambiguous test sentence with an unambiguous control sentence.

The benchmark evaluates two primary types of failures:

  1. Explicit Errors: Observed in word segmentation models that choose the incorrect boundary.
  2. Implicit Errors: Observed in character-level sentiment analysis models (like BERT) where the sentiment of a non-existent "canary word" leaks into the final prediction, skewing the model's output despite no explicit segmentation step.

To confirm that these errors stem specifically from the garden path, the authors conducted occlusion studies, masking the canary word characters to see if model performance recovered.

Key Insights and Results

The study revealed several critical vulnerabilities in current Chinese NLP models:

  • Left-Branching Bias: Most segmentation models exhibit a "greedy" preference for grouping the first two characters of a trigram. This suggests that models rely heavily on local character frequency rather than global syntactic or semantic context.
  • Sentiment Leakage: Even "segmentation-free" models are vulnerable. The sentiment of an incorrect canary word can significantly alter a model's score. Off-the-shelf models like RoBERTa-110M showed Garden Path Error Rates (GPER) as high as 45.4%.
  • Contextual Failure: Despite the long-range dependency capabilities of Transformers, models frequently ignore the morphosyntactic context that should make the correct parse obvious to a human reader.
  • Human-Model Gap: Humans achieved 91.3% accuracy on these tasks, proving that these garden paths are easily resolvable by native speakers. In contrast, even the best-performing models lagged significantly.

Mitigation and Future Work

The researchers found that injecting word boundary information can mitigate these errors. Strategies such as Whole Word Masking (WWM) during pre-training or Joint Training with word segmentation tasks significantly improved robustness. A combined WWM+CWS model achieved the lowest GPER of 18.8%.

While ERAS provides a large-scale (203,944 pairs), controlled environment for testing, its synthetic nature is a limitation. Future work may involve extending this analysis to more diverse natural language structures and other NLP tasks like Named Entity Recognition (NER) or Machine Translation. Furthermore, evaluating closed-source Large Language Models (LLMs) remains a challenge, as current metrics often require access to internal model logits.

Summary updated

BibTeX
@inproceedings{li2025eras,
  author = {Qinchan Li and Sophie Hao},
  title = {{ERAS: Evaluating the Robustness of Chinese NLP Models to Morphological Garden Path Errors}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.159},
}
Figure