GLS-CSC: A Simple but Effective Strategy to Mitigate Chinese STM Models' Over-Reliance on Superficial Clue

Yanrui Du, Sendong Zhao, Yuhan Chen, Rai Bai, Jing Liu, Hua Wu, Haifeng Wang, Bing QinarXiv 2023 · DOI · arXiv

Introduction

Short Text Matching (STM) is a fundamental task in Natural Language Processing (NLP), critical for applications such as semantic search, question answering, and conversational AI. The goal is to determine the semantic relationship—typically "matched" or "mismatched"—between two short text segments. While large-scale pre-trained language models (PLMs) like BERT, ERNIE, and RoBERTa have achieved state-of-the-art performance on standard benchmarks, they often exhibit a tendency to rely on "superficial clues" rather than genuine semantic understanding.

Comparison between model prediction based on literal similarity and human judgment based on semantics.

In Chinese STM, a particularly potent superficial clue is edit distance. Because Chinese is a highly sequential language where minor character changes or reordering can drastically alter meaning, models often learn a statistical shortcut: low edit distance implies a match, while high edit distance implies a mismatch. This over-reliance makes models brittle, leading to failures on adversarial examples or when deployed in specialized domains like finance or medicine where these shortcuts may not hold.

The Superficial Clue: Edit Distance

The authors identify that models are "lazy" learners. If a simple statistical feature like edit distance provides high accuracy on the training set, the model will prioritize this shortcut over complex semantic reasoning. To quantify this, the researchers analyzed the label distribution across various edit distances in common datasets.

Label distribution as a function of edit distance, showing strong correlation between low distance and 'Match' (Label 1) and high distance and 'Mismatch' (Label 0).

As shown in Figure 2, there is a clear crossover point. When the edit distance is very small (e.g., 1-3), the probability of the label being "Matched" (Label 1) is extremely high. Conversely, as the edit distance increases, the probability of a "Mismatched" (Label 0) label dominates. Models exploit this correlation, effectively "cheating" during training and failing to generalize when the semantic meaning contradicts the literal similarity.

The GLS-CSC Strategy

To combat this, the paper introduces GLS-CSC (Gradually Learn Samples Containing Superficial Clue). This resampling strategy is designed to force the model to prioritize deep semantic reasoning by controlling the exposure to "easy" shortcut-heavy samples.

The methodology consists of four key steps:

  1. Identification of CSC-Samples: Samples where the label is strongly correlated with edit distance are flagged as "Samples Containing Superficial Clues" (CSC-Samples).
  2. Easy vs. Hard Categorization: Data is partitioned into "easy-to-predict" samples (where the shortcut leads to the correct answer) and "hard-to-predict" samples (where the shortcut leads to an error).
  3. Gradual Resampling: Instead of random shuffling, GLS-CSC starts training by focusing on "hard" samples that require genuine linguistic logic. The proportion of "easy" CSC-Samples is then increased linearly throughout the training process.
  4. Linear Growth Schedule: The authors found that a gradual introduction is superior to a sudden "dump" of easy samples at the end of training. A sudden introduction (LLS-CSC) can cause a sharp drop in training loss that prevents the model from maintaining its learned reasoning capabilities, whereas linear growth ensures stable integration.

Experimental Evaluation

The researchers evaluated GLS-CSC across several Chinese datasets, including LCQMC and CCKS for general training, and the DuQM dataset for robustness testing. They also tested generalization on out-of-domain (O.O.D.) datasets from the finance (BQ), medical (CHIP-STS), and dialogue (OPPO) sectors.

The results demonstrate that GLS-CSC consistently outperforms standard fine-tuning and other debiasing techniques like Product-of-Experts (PoE). On the DuQM robustness test, GLS-CSC improved BERT's accuracy by 5.13%. Furthermore, it provided a steady improvement of up to 1.10% across O.O.D. sets, whereas other methods often saw performance regressions in specific domains. Interpretability analysis using LIME confirmed that GLS-CSC models focus on key semantic differences rather than just overlapping characters.

Conclusion

GLS-CSC offers a simple yet powerful way to improve the robustness of Chinese STM models. By strategically managing the curriculum of "shortcut" samples, it prevents models from becoming overly reliant on superficial clues like edit distance. While the current approach requires manual identification of the relevant shortcuts, it paves the way for future research into automated shortcut detection and mitigation across a broader range of NLP tasks.

Summary updated

Abstract

Pre-trained models have achieved success in Chinese Short Text Matching (STM) tasks, but they often rely on superficial clues, leading to a lack of robust predictions. To address this issue, it is crucial to analyze and mitigate the influence of superficial clues on STM models. Our study aims to investigate their over-reliance on the edit distance feature, commonly used to measure the semantic similarity of Chinese text pairs, which can be considered a superficial clue. To mitigate STM models' over-reliance on superficial clues, we propose a novel resampling training strategy called Gradually Learn Samples Containing Superficial Clue (GLS-CSC). Through comprehensive evaluations of In-Domain (I.D.), Robustness (Rob.), and Out-Of-Domain (O.O.D.) test sets, we demonstrate that GLS-CSC outperforms existing methods in terms of enhancing the robustness and generalization of Chinese STM models. Moreover, we conduct a detailed analysis of existing methods and reveal their commonality.
BibTeX
@misc{du2023gls,
  author = {Yanrui Du and Sendong Zhao and Yuhan Chen and Rai Bai and Jing Liu and Hua Wu and Haifeng Wang and Bing Qin},
  title = {{GLS-CSC: A Simple but Effective Strategy to Mitigate Chinese STM Models' Over-Reliance on Superficial Clue}},
  year = {2023},
  eprint = {2309.04162},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.04162},
}
Figure