TL;DR
This paper introduces "2kenize," a novel subword segmentation model that jointly considers both Simplified Chinese (SC) and Traditional Chinese (TC) to improve script conversion accuracy. By addressing the ambiguity of one-to-many character mappings and the challenges of Chinese word segmentation, 2kenize achieves a significant 6-point accuracy improvement over existing methods and demonstrates its effectiveness in downstream tasks like topic classification.
What Problem Does This Paper Address?
Chinese Natural Language Processing (NLP) often requires converting text between Simplified Chinese (SC) and Traditional Chinese (TC). While SC to TC conversion is relatively straightforward, the reverse, TC to SC, is more complex due to the fact that a single SC character can map to multiple TC characters depending on the context. Existing methods struggle with this ambiguity, leading to suboptimal performance, especially with code-mixing (mixing of different languages or dialects within a text) and named entities. Furthermore, Chinese is an unsegmented language, meaning words are not explicitly separated by spaces, making tokenization a critical and challenging preprocessing step that significantly impacts conversion accuracy. The authors also highlight a lack of comprehensive benchmark datasets for evaluating these conversion tasks.
Methods
The core of the proposed solution is "2kenize," a subword segmentation model designed to jointly process SC and TC. Instead of treating segmentation and conversion as separate monolingual tasks, 2kenize integrates them. The model leverages:
- Subword Segmentation: This approach breaks down words into smaller units, which helps in handling rare words and improving generalization.
- Two Language Models: One for SC and one for TC, used to score the likelihood of different segmentations and conversions.
- Viterbi Tokenizer: A dynamic programming algorithm that finds the most likely sequence of subword mappings.
- Lookahead Mechanism: 2kenize incorporates a "lookahead" feature, allowing it to consider potential target TC sequences while segmenting the source SC sentence. This helps in disambiguating one-to-many mappings.
- Beam Search: Used to find the best possible TC sequence from the generated mapping sequences.
A variant, "1kenize," is also introduced, which focuses on tokenization for TC sentences and is applied to downstream tasks.
Datasets
The authors introduce new benchmark datasets for both intrinsic evaluation (script conversion) and extrinsic evaluation (topic classification):
- Intrinsic Evaluation Datasets: Gold standard corpora were constructed for both SC and TC, spanning four domains: Hong Kong Literature, Hong Kong News, Taiwanese Literature, and Taiwanese Newswire. These datasets contain approximately 3000 sentences each and were created by heuristically converting TC sentences to SC using OpenCC and then manually correcting them.
- Extrinsic Evaluation Datasets:
- Pretraining Dataset: Chinese Wikipedia articles were used for pretraining language models.
- Topic Classification Datasets: * Formal Text Classification: News articles scraped from Singtao (2017-2019) covering financial, educational, local, international, and sports categories.
- Informal Text Classification: Social media posts scraped from LIHKG (2017-2018) covering sports, opinions, memes, IT, financial, and leisure categories.
Key Insights and Ideas
- Joint Segmentation and Conversion is Crucial: The paper strongly advocates for treating script conversion not as a simple dictionary lookup but as a task that requires joint consideration of subword segmentation and language modeling across both scripts.
- Subword Ambiguity is a Strength: Unlike some methods that treat subword ambiguity as noise, 2kenize leverages it by considering multiple possible segmentations, leading to better disambiguation.
- Context is King for Ambiguous Characters: The core challenge lies in one-to-many mappings. 2kenize's ability to "look ahead" at potential TC conversions while segmenting SC text is key to resolving these ambiguities.
- Code-Mixing and Named Entities are Strengths: The model shows particular proficiency in handling code-mixed text and named entities, areas where previous methods often faltered.
- Subword Regularization Benefits Downstream Tasks: Applying 2kenize to pretraining datasets for topic classification significantly boosted classifier performance, demonstrating the practical utility of improved script conversion.
Key Results
- Intrinsic Evaluation: 2kenize achieved a sentence accuracy of 92.4% and a disambiguation error density (DED) of 3.0, outperforming the best existing converter, OpenCC (DED 4.3, SA 85.3%), by a significant margin.
- Qualitative Analysis: Error analysis revealed that 2kenize excels at handling code-mixing (Cantonese and English) and disguised named entities, where other methods struggled.
- Extrinsic Evaluation (Topic Classification): When 2kenize was used to convert pretraining datasets, it led to an absolute improvement of 0.9% on formal and 4.5% on informal topic classification tasks compared to other converters.
- Tokenization Impact: The paper also demonstrated that LM-based subword tokenization (like 2kenize) produces less skewed token distributions, leading to better performance on noisy datasets compared to character or word-based tokenization.
Limitations and Future Work
- Limitations:
- While 2kenize shows strong performance, the paper acknowledges that some errors are still caused by infrequent characters or variant characters used interchangeably.
- The performance on news text was slightly worse due to a higher number of new entities.
- The authors note that seq2seq approaches are generally unsuitable for this task due to their tendency to reword the target sentence.
- Future Work:
- The authors pose questions about exploiting subword variations to reduce skewness in NLU tasks.
- They also wonder if subword-segmentation-transfer could be beneficial for other NMT-NLU task pairs, similar to how 2kenize was used for script conversion and then 1kenize for classification.
Summary updated