2kenize: Tying Subword Sequences for Chinese Script Conversion

A Pranav, Isabelle Augenstein

TL;DR

This paper introduces "2kenize," a novel subword segmentation model for Chinese script conversion that tackles the ambiguity of one-to-many character mappings between Simplified and Traditional Chinese. By jointly considering source and target scripts, 2kenize significantly improves conversion accuracy and demonstrates its effectiveness in downstream tasks like topic classification, particularly excelling with code-mixed and named entity text.

What Problem Does This Paper Address?

Chinese Natural Language Processing (NLP) often requires converting text between Simplified Chinese (SC) and Traditional Chinese (TC). While seemingly straightforward, this conversion is complicated by the fact that a single SC character can map to multiple TC characters depending on context. Existing methods, often relying on simple dictionary lookups or basic language models, struggle with this ambiguity, leading to suboptimal performance. This under-addressed problem limits the effectiveness of Chinese NLP applications, especially for TC users who have less access to resources compared to SC users. The paper aims to bridge this gap by proposing a more sophisticated approach that can accurately disambiguate these character mappings.

Methods

The core of the proposed solution is 2kenize, a model that leverages subword segmentation and joint language modeling to perform Chinese script conversion. Unlike traditional methods that process scripts independently, 2kenize considers both the source SC sentence and its potential TC conversions simultaneously.

Here's a breakdown of the key methodological components:

  • Subword Segmentation: The model breaks down text into subword units. This is a standard technique in NLP for handling rare words and improving generalization, but 2kenize applies it in a novel way for script conversion.
  • Joint Language Modeling: 2kenize employs two language models (LMs) to understand the context of both SC and TC. This allows it to better disambiguate mappings by considering the likelihood of different TC sequences.
  • Viterbi Tokenizer: A Viterbi algorithm is used to find the most probable sequence of subword mappings. This dynamic programming approach efficiently explores possible conversions.
  • Mapping Sequence Modeling: The model learns to map subword sequences from SC to TC, considering the probabilities of these mappings.
  • Beam Search: After identifying potential mapping sequences, beam search is used to find the best overall TC sentence.

A variant called 1kenize is also introduced, which focuses on tokenization for TC sentences and improves accuracy on topic classification tasks.

Datasets

The paper introduces new benchmark datasets and utilizes existing ones for training and evaluation:

  • New Datasets:
    • Character Conversion Evaluation Datasets: Spanning Hong Kong and Taiwanese literature and news genres.
  • Traditional Chinese Topic Classification Datasets: Formal (scraped from Singtao) and informal (scraped from LIHKG) styles, covering news, social media discussions, and memes.
  • Existing Datasets Used for Training:
    • SIGHAN-2005 Bakeoff dataset: Used for training the segmentation-based language model (PKU and MSR partitions for SC, Academia Sinica and CityU partitions for TC).
  • Chinese Wikipedia: Used for pretraining language models.
  • Wikitext: Used for downstream topic classification tasks.

Key Insights and Ideas

  • The Ambiguity of SC-TC Mapping is Crucial: The paper highlights that a significant portion (12%) of SC characters have one-to-many mappings to TC characters, making simple dictionary-based conversion insufficient.
  • Joint Modeling is Key: The core insight is that processing SC and TC scripts jointly, rather than independently, leads to significantly better disambiguation and conversion accuracy.
  • Subword Segmentation for Disambiguation: Applying subword segmentation in conjunction with language models allows the model to better handle context and resolve ambiguities that character-level models miss.
  • Impact on Downstream Tasks: The quality of script conversion directly impacts the performance of downstream NLP tasks. By improving conversion, 2kenize enhances the utility of datasets for tasks like topic classification.
  • Handling Code-Mixing and Named Entities: The model shows particular strength in dealing with complex text containing vernacular Cantonese (zh-yue), English, and named entities, areas where other methods struggle.

Key Results

  • Improved Accuracy: 2kenize achieves a 6-point improvement in sentence accuracy compared to previous Chinese character conversion approaches.
  • Benchmark Performance: The model attains an average Disambiguation Error Density (DED) of 3.0 and a Sentence Accuracy (SA) of 92.4% overall, outperforming OpenCC (DED 4.3, SA 85.3%).
  • Downstream Task Improvement: When used to convert pretraining datasets for topic classification, 2kenize leads to improved classifier performance, with an absolute improvement of 0.9% on formal and 4.5% on informal topic classification tasks compared to other converters.
  • Qualitative Strengths: Error analysis reveals 2kenize's particular strengths in handling code mixing (e.g., with Cantonese and English) and named entities.
  • Tokenization Impact: The study demonstrates that subword tokenizers, especially context-aware ones like 2kenize, produce less skewed token distributions, leading to better performance on noisy datasets.

Limitations and Future Work

  • Limitations:
    • The paper notes that while 2kenize performs well, some errors are still caused by infrequent characters or variant characters.
  • The performance on news text is slightly worse due to the high number of new entities.
  • Seq2seq approaches are deemed unsuitable for this task due to their tendency to reword the target sentence.
  • Future Work:
    • The authors suggest exploring how to exploit subword variations to reduce skewness in Natural Language Understanding (NLU) tasks.
  • They also propose investigating whether subword-segmentation-transfer could be beneficial for other NMT-NLU task pairs, similar to how script conversion was used for classification.

Summary updated

Abstract

Simplified Chinese to Traditional Chinese character conversion is a common preprocessing step in Chinese NLP. Despite this, current approaches have insufficient performance because they do not take into account that a simplified Chinese character can correspond to multiple traditional characters. Here, we propose a model that can disambiguate between mappings and convert between the two scripts. The model is based on subword segmentation, two language models, as well as a method for mapping between subword sequences. We further construct benchmark datasets for topic classification and script conversion. Our proposed method outperforms previous Chinese Character conversion approaches by 6 points in accuracy. These results are further confirmed in a downstream application, where 2kenize is used to convert pretraining dataset for topic classification. An error analysis reveals that our method’s particular strengths are in dealing with code mixing and named entities.
BibTeX
@inproceedings{pranav2020kenize,
  author = {A Pranav and Isabelle Augenstein},
  title = {{2kenize: Tying Subword Sequences for Chinese Script Conversion}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.648},
}
Figure