Abstract
Chinese Spell Check (CSC) faces significant challenges due to massive vocabulary sizes, data sparsity, and domain shift between input mediums. This paper introduces HeadFilt, an adaptable filtering layer that replaces rigid, handcrafted confusion sets with learned hierarchical character embeddings. By leveraging the internal radical structure of Chinese characters and fine-tuning on domain-specific errors, the model achieves state-of-the-art results and demonstrates robustness against shifting error patterns between handwritten and typed text.
Introduction
Chinese Spell Check is notoriously difficult for three primary reasons. First, the massive vocabulary exceeds 10,000 common characters. Second, data sparsity leads to "long-tail" error patterns where over 80% of error types appear only once in typical datasets. Finally, domain shift occurs because error patterns change based on the input method; handwritten text often involves visual confusion, while Pinyin-based typing leads to phonetic confusion.
Traditional systems rely on "confusion sets"—static, handcrafted lists of likely character swaps. However, these sets lack flexibility. When the domain shifts (e.g., from 2013 handwritten data to 2015 typed data), fixed sets lose coverage, failing to detect nearly 25% of errors.
Methodology: The HeadFilt Pipeline
The proposed system utilizes a two-stage pipeline to address these challenges:
- The Generator: A standard pre-trained Masked Language Model (BERT) predicts a probability distribution over the entire vocabulary for a given character position.
- The Filter (HeadFilt): This core innovation calculates a similarity vector between the input character and the rest of the vocabulary. This vector is multiplied element-wise with BERT’s output to "filter out" improbable corrections.
Hierarchical Embeddings and TreeLSTM
What makes HeadFilt unique is its use of a TreeLSTM to encode the actual structure of Chinese characters, including radicals and components. This creates an "inductive bias" where characters sharing physical components are naturally closer in the embedding space. Unlike static lists, HeadFilt is trained using a contrastive loss, learning to pull confusable characters together and push dissimilar ones apart.
Adaptable Training
HeadFilt can be fine-tuned on new datasets to adapt to shifting error patterns. This allows the model to learn new phonetic confusions common in modern Input Method Editors (IMEs) without requiring manual updates to a confusion set.
Evaluation and Results
The model was evaluated on the SIGHAN Chinese Spelling Check Bake-off datasets (2013–2015). The 2013 dataset features handwritten essays, while 2014 and 2015 feature typed text. The overlap of error types between these years is as low as 1%, making them ideal for testing domain-shift robustness.
Key Performance Metrics
- State-of-the-Art Results: HeadFilt achieved SOTA results on the 2014 and 2015 datasets for error correction.
- Superiority over Fixed Sets: On the 2015 dataset, HeadFilt achieved an F1 score of 0.690, significantly outperforming the "FixedFilt" baseline (0.658).
- The Recall Paradox: In error detection, filtering can lower recall by being more conservative. However, in error correction, it significantly increases recall. By narrowing the search space to plausible candidates, the model avoids being distracted by grammatically correct but contextually incorrect "false positives."
Conclusion and Future Work
HeadFilt automates the creation of confusion sets, allowing systems to self-correct internal similarity maps as they encounter new data. Future research may focus on explicitly incorporating phonetic systems like Pinyin or Zhuyin to further improve accuracy for typed text and addressing contextual ambiguity in heteronyms (characters with multiple pronunciations). The authors also note the potential for this structural approach to benefit other logographic languages such as Japanese (Kanji) or Korean (Hanja).
Summary updated