Two Sequence Labeling Approaches to Sentence Segmentation and Punctuation Prediction for Classic Chinese Texts

Xuebin Wang, Zhenghua LiCOLING 2024

TL;DR

This paper tackles the challenge of automatically segmenting sentences and predicting punctuation in classical Chinese texts, which historically lack explicit punctuation. The authors propose two sequence labeling approaches, a one-stage and a two-stage method, both leveraging the SikuRoBERTa pre-trained model. Their two-stage approach, which distinguishes between "pause" and "non-pause" punctuation marks, achieved second place in the EvaHan2024 shared task, demonstrating superior performance over the one-stage method.

What Problem Does This Paper Address?

Classical Chinese texts present a unique challenge for natural language processing due to their historical lack of punctuation marks. This absence makes automatic sentence segmentation (SS) and punctuation prediction (PP) significantly more ambiguous compared to modern languages. Readers must infer sentence boundaries and punctuation based on context and linguistic cues. The EvaHan2024 shared task aimed to develop computational models capable of performing these tasks, bridging the gap between historical texts and modern NLP capabilities.

Methods

The authors explore two distinct sequence labeling approaches:

  • One-Stage Approach: This method directly predicts a label for each character. A single label can encompass multiple punctuation marks, simplifying the output structure but potentially increasing label complexity.
  • Two-Stage Approach: This approach first categorizes punctuation marks into two types: "pause" marks (indicating sentence boundaries, like periods and exclamation marks) and "non-pause" marks (other punctuation). These two types are then handled by two separate sequence labeling models. Each label in this approach contains at most one punctuation mark. This distinction allows the SS task to be treated as a byproduct of PP.

Both approaches utilize a SikuRoBERTa model as the encoder, a pre-trained language model specifically designed for traditional Chinese texts. A Conditional Random Field (CRF) layer is applied on top of the encoder to refine the predictions. The model architecture involves character embeddings concatenated with SikuRoBERTa embeddings, followed by three BiLSTM layers for contextual representation, and finally an MLP layer before the CRF.

Datasets

  • Training Data: The primary training data is derived from the Siku Quanshu, a vast collection of classical Chinese texts, comprising over 10 million characters.
  • Provisional Validation Data: 5% of the training data was set aside for validation.
  • Synthetic Data: The authors generated approximately 11,000 synthetic classical Chinese sentences using the Xunzi-Qianwen-7B-CHAT model to augment training and validation.
  • Test Set: The evaluation was performed on a test set, with results presented for two scenarios: "Test A" (general) and "Test B" (specifically on the Zuozhuan test set).

Key Insights and Ideas

  • The Power of Distinction: The core insight is that distinguishing between "pause" and "non-pause" punctuation marks significantly improves performance. This categorization directly aids in sentence segmentation and allows for a more focused approach to punctuation prediction.
  • SikuRoBERTa's Advantage: Using SikuRoBERTa, a model pre-trained on traditional Chinese texts, proved more effective than models pre-trained on simplified Chinese for this task, highlighting the importance of domain-specific pre-training.
  • Two-Stage Superiority: The two-stage approach, despite its apparent complexity, outperformed the one-stage approach. This suggests that breaking down the problem into more manageable, specialized sub-tasks can lead to better overall results.
  • CRF for Refinement: The integration of a CRF layer on top of the deep learning model effectively captures dependencies between labels, leading to more coherent and accurate predictions.

Key Results

The paper reports strong performance on the EvaHan2024 shared task:

  • The two-stage approach achieved second place among all participating systems.
  • On the Zuozhuan test set (Test B), the two-stage model achieved an F1 score of 93.27% for Sentence Segmentation and 80.89% for Punctuation Prediction.
  • The two-stage model consistently outperformed the one-stage model across both evaluation sets and tasks.
  • The proposed models showed a significant improvement compared to the baseline model (Xunzi-Qianwen-7B-CHAT).

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Two-Stage Method Elegance: The current two-stage method's post-processing for resolving order conflicts between predicted punctuation marks is described as "not elegant." They propose training a binary classifier to determine the order as a more refined alternative.
  • Data Processing Issues: Incomplete data processing due to the expansive nature and encoding difficulties of traditional Chinese characters led to missing characters or incomplete sentences. These were treated as noise and removed, but a more robust handling mechanism is desired.
  • Rule-Based Correction Limitations: While a rule-based method was used to correct illegal punctuation marks, its efficacy is acknowledged as limited.
  • BiLSTM Speed: The BiLSTM layers process lengthy texts slowly, contributing to a longer training process.
  • XunziALLM Underutilization: The full potential of the XunziALLM tool for data generation was not completely leveraged.
  • Future Work: Exploring more elegant methods for handling multi-stage predictions and improving data processing for challenging classical Chinese texts are key areas for future research.

Summary updated

BibTeX
@inproceedings{wang2024two,
  author = {Xuebin Wang and Zhenghua Li},
  title = {{Two Sequence Labeling Approaches to Sentence Segmentation and Punctuation Prediction for Classic Chinese Texts}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure