Active Learning for Chinese Word Segmentation

Shoushan Li, Guodong Zhou, Chu-Ren HuangCOLING 2012

TL;DR

This paper tackles the high cost of annotating data for Chinese Word Segmentation (CWS) by proposing an active learning approach that focuses on annotating only uncertain word boundaries, significantly reducing manual effort. They introduce a novel Word Boundary Annotation (WBA) model and a diversity measurement to avoid redundant annotations, achieving substantial cost savings with minimal performance loss.

What Problem Does This Paper Address?

Chinese Word Segmentation (CWS) is a crucial preprocessing step for many natural language processing tasks involving Chinese text. However, state-of-the-art CWS models require massive amounts of pre-segmented data for training, which is extremely time-consuming and expensive to acquire. Active learning offers a promising solution by intelligently selecting the most informative data points for human annotation, thereby minimizing the overall annotation cost. The challenge in applying active learning to CWS lies in its inherent nature: unlike traditional classification tasks where an instance is a single entity, CWS treats each character's position within a word as a label. This makes it difficult to define what constitutes an "informative instance" for annotation. Furthermore, the sheer scale of CWS datasets (millions of characters) makes traditional instance-level active learning computationally prohibitive.

Methods

The authors propose a novel Word Boundary Annotation (WBA) model as the core of their active learning strategy for CWS. Instead of annotating entire sentences or characters, WBA focuses on annotating only the uncertain word boundaries between adjacent characters. This means that for a given character sequence, only the boundaries that the current model is most unsure about are presented to the human annotator. The surrounding characters and their boundaries are automatically labeled.

To further optimize the annotation process and address the issue of redundant annotations, they introduce a diversity measurement. This measure helps to avoid selecting multiple instances that convey similar information. Specifically, they leverage the frequency of surrounding character bigrams to quantify the diversity of a boundary. By combining uncertainty and diversity, their approach ranks potential annotation candidates, prioritizing those that are both uncertain and unique.

The overall active learning framework involves iteratively:

  1. Training a CWS segmenter using the current labeled data.
  2. Using this segmenter to predict labels for all unlabeled boundaries.
  3. Employing the uncertainty-diversity sampling strategy to select a batch of the most informative and diverse boundaries for manual annotation.
  4. Adding these newly annotated boundaries (along with their automatically labeled context) to the labeled set for the next iteration.

Datasets

The experiments were conducted on the SIGHAN Bakeoff 2 dataset, which consists of four corpora: PKU, MSR, CityU, and AS. Due to its large scale, the AS corpus was excluded from the experiments. The paper also describes an experiment where they randomly draw three data sets from the PKU training data, each containing 50 sentences, for a manual annotation effort study.

Key Insights and Ideas

  • Focusing on Boundaries is Key: The core insight is that for CWS, the most efficient annotation unit is not a character or a sentence, but the uncertain word boundary. This drastically reduces the annotation burden compared to annotating entire sentences.
  • Diversity Prevents Redundancy: Simply selecting the most uncertain instances can lead to annotating similar information multiple times. Incorporating a diversity measure, based on character bigram frequencies, effectively avoids this redundancy and further saves annotation effort.
  • Dynamic Diversity Measurement: The proposed diversity measure is dynamic, meaning the diversity value of a boundary can change during training as more data is processed. This allows the system to adapt and prioritize novel information.
  • Significant Cost Savings: The active learning approach, particularly with the WBA model and diversity measurement, demonstrably saves a substantial amount of annotation time and effort, averaging an 85% reduction in annotation time in one experiment.

Key Results

The paper reports that their proposed uncertainty-diversity sampling strategy consistently outperforms both random selection and uncertainty sampling alone. In experiments on the SIGHAN Bakeoff 2 dataset, this strategy achieved significantly better performance (measured by F1-score) with a given proportion of annotated boundaries. For instance, the word "企業/enterprise" appeared 392 times in data annotated using uncertainty sampling, but only 144 times when uncertainty-diversity sampling was used, highlighting the effectiveness of diversity in avoiding duplicate annotations.

In the annotation effort experiment, their WBA approach (annotating only uncertain boundaries) saved an average of 85% of annotation time compared to annotating all sentences, and 37.4% compared to annotating sentences containing uncertain boundaries.

Limitations and Future Work

The paper acknowledges that the AS corpus from SIGHAN Bakeoff 2 was excluded due to its large scale, which could cause out-of-memory errors. This suggests that further optimization might be needed for extremely large datasets.

Future work could explore more sophisticated diversity measures or investigate how to further automate the labeling of context boundaries. The paper also implicitly suggests that the WBA model and diversity measurement could be beneficial for other sequence labeling tasks where annotation is costly.

Summary updated

BibTeX
@inproceedings{li2012active,
  author = {Shoushan Li and Guodong Zhou and Chu-Ren Huang},
  title = {{Active Learning for Chinese Word Segmentation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2012},
}
Figure