Introduction
Short Text Matching (STM) is a fundamental task in Natural Language Processing (NLP), critical for applications such as semantic search, question answering, and conversational AI. The goal is to determine the semantic relationship—typically "matched" or "mismatched"—between two short text segments. While large-scale pre-trained language models (PLMs) like BERT, ERNIE, and RoBERTa have achieved state-of-the-art performance on standard benchmarks, they often exhibit a tendency to rely on "superficial clues" rather than genuine semantic understanding.
In Chinese STM, a particularly potent superficial clue is edit distance. Because Chinese is a highly sequential language where minor character changes or reordering can drastically alter meaning, models often learn a statistical shortcut: low edit distance implies a match, while high edit distance implies a mismatch. This over-reliance makes models brittle, leading to failures on adversarial examples or when deployed in specialized domains like finance or medicine where these shortcuts may not hold.
The Superficial Clue: Edit Distance
The authors identify that models are "lazy" learners. If a simple statistical feature like edit distance provides high accuracy on the training set, the model will prioritize this shortcut over complex semantic reasoning. To quantify this, the researchers analyzed the label distribution across various edit distances in common datasets.
As shown in Figure 2, there is a clear crossover point. When the edit distance is very small (e.g., 1-3), the probability of the label being "Matched" (Label 1) is extremely high. Conversely, as the edit distance increases, the probability of a "Mismatched" (Label 0) label dominates. Models exploit this correlation, effectively "cheating" during training and failing to generalize when the semantic meaning contradicts the literal similarity.
The GLS-CSC Strategy
To combat this, the paper introduces GLS-CSC (Gradually Learn Samples Containing Superficial Clue). This resampling strategy is designed to force the model to prioritize deep semantic reasoning by controlling the exposure to "easy" shortcut-heavy samples.
The methodology consists of four key steps:
- Identification of CSC-Samples: Samples where the label is strongly correlated with edit distance are flagged as "Samples Containing Superficial Clues" (CSC-Samples).
- Easy vs. Hard Categorization: Data is partitioned into "easy-to-predict" samples (where the shortcut leads to the correct answer) and "hard-to-predict" samples (where the shortcut leads to an error).
- Gradual Resampling: Instead of random shuffling, GLS-CSC starts training by focusing on "hard" samples that require genuine linguistic logic. The proportion of "easy" CSC-Samples is then increased linearly throughout the training process.
- Linear Growth Schedule: The authors found that a gradual introduction is superior to a sudden "dump" of easy samples at the end of training. A sudden introduction (LLS-CSC) can cause a sharp drop in training loss that prevents the model from maintaining its learned reasoning capabilities, whereas linear growth ensures stable integration.
Experimental Evaluation
The researchers evaluated GLS-CSC across several Chinese datasets, including LCQMC and CCKS for general training, and the DuQM dataset for robustness testing. They also tested generalization on out-of-domain (O.O.D.) datasets from the finance (BQ), medical (CHIP-STS), and dialogue (OPPO) sectors.
The results demonstrate that GLS-CSC consistently outperforms standard fine-tuning and other debiasing techniques like Product-of-Experts (PoE). On the DuQM robustness test, GLS-CSC improved BERT's accuracy by 5.13%. Furthermore, it provided a steady improvement of up to 1.10% across O.O.D. sets, whereas other methods often saw performance regressions in specific domains. Interpretability analysis using LIME confirmed that GLS-CSC models focus on key semantic differences rather than just overlapping characters.
Conclusion
GLS-CSC offers a simple yet powerful way to improve the robustness of Chinese STM models. By strategically managing the curriculum of "shortcut" samples, it prevents models from becoming overly reliant on superficial clues like edit distance. While the current approach requires manual identification of the relevant shortcuts, it paves the way for future research into automated shortcut detection and mitigation across a broader range of NLP tasks.
Summary updated