Handling Cross- and Out-of-Domain Samples in Thai Word Segmentation

Peerat Limkonchotiwat, Wannaphong Phatthiyaphaibun, Raheem Sarwar, Ekapol Chuangsuwanich, Sarana NutanongACL 2021

TL;DR

This paper tackles the challenge of Thai word segmentation, which is difficult due to its continuous script and domain-dependent nature. The authors propose a novel framework combining deep learning with stacked ensembles (DSE) for domain adaptation and a Multi-Domain Ensemble (MDE) for handling out-of-domain inputs, achieving significant performance improvements across multiple languages.

What Problem Does This Paper Address?

Word segmentation (WS) is a fundamental preprocessing step for many Natural Language Processing (NLP) tasks, including named entity recognition and machine translation. While WS is relatively solved for languages with clear word boundaries, it remains a significant challenge for continuous-script languages like Thai, Chinese, and Japanese, where words are not explicitly separated. Furthermore, WS models are often domain-dependent, meaning a model trained on one type of text (e.g., news articles) may perform poorly on another (e.g., social media). This is particularly problematic for low-resource languages like Thai, where obtaining sufficient domain-specific training data is difficult. The paper addresses two key issues: (1) adapting existing WS models to new domains with limited data, and (2) handling inputs from entirely unseen domains (out-of-domain scenarios).

Methods

The authors introduce two main methodological contributions:

  • Deep Stacked Ensemble (DSE) for Domain Adaptation: Building upon the concept of stacked ensembles (SE), they propose DSE, which enhances the domain-specific component of the SE model with a deep learning architecture (Bi-LSTM with Attention). This deep learning component leverages character n-grams, character type n-grams, and features from the domain-generic model (probability and entropy) to better capture intricate WS patterns in the target domain. They also introduce a data augmentation technique using Masked Language Models (MLM) to generate "hard-to-segment" and "semi-hard-to-segment" samples, effectively increasing the training data for low-resource scenarios.
  • Multi-Domain Ensemble (MDE) for Out-of-Domain Scenarios: To handle inputs from completely unseen domains, they propose MDE. This framework utilizes an ensemble of multiple domain-specific models, each adapted to a particular domain. These models' predictions are then aggregated using a result aggregation module. They explore two aggregation strategies: simple averaging (Avg) and minimum entropy (ME), finding ME to be more effective.

Datasets

The paper uses several benchmark datasets for evaluation:

  • Thai Word Segmentation (TWS) Corpora:
    • Wisesight (social media)
    • TNHC (classical literature)
    • LST20 (generic)
    • VISTEC (newly introduced social media dataset)
  • New Dataset: The authors introduce "VISTEC-TP-TH-2021", a social media dataset derived from Twitter (2017-2019) containing 49,997 sentences. This dataset is annotated for four tasks: word segmentation, named-entity boundary detection, and misspelling detection and correction. It was specifically curated to address the underrepresentation of social media data in existing benchmarks.
  • Cross-lingual Datasets:
    • Chinese: AS, CITYU (CI), MSR (MS), PKU (PK)
    • Japanese: GSD (GS), Modern (MO), PUD (PU)
    • Urdu: UCRF (UC)

Key Insights and Ideas

  • Deep Learning Enhances Stacked Ensembles: Simply replacing the traditional domain-specific component in stacked ensembles with a deep learning model (Bi-LSTM with Attention) significantly boosts performance in domain adaptation.
  • Data Augmentation is Crucial for Low-Resource Domains: For domains with limited data, data augmentation techniques, particularly those generating "hard-to-segment" samples using MLM, are essential for improving model performance.
  • Ensembling for Robustness: An ensemble of domain-specific models (MDE) provides a robust solution for handling out-of-domain inputs, outperforming single models or simple transfer learning approaches.
  • Minimum Entropy Aggregation is Key: The minimum entropy aggregation strategy for MDE proved more effective than simple averaging, suggesting that models with higher confidence (lower entropy) are more reliable for out-of-domain predictions.
  • Generalizability Across Languages: The proposed DSE and MDE frameworks demonstrate significant improvements not only for Thai but also for Chinese, Japanese, and Urdu, highlighting their broad applicability.

Key Results

  • Thai Domain Adaptation: DSE improved Thai word segmentation performance from 93.47% to 96.67% (character level) and 84.03% to 91.51% (word level) compared to the state-of-the-art DC model. With data augmentation, further improvements were observed.
  • Thai Out-of-Domain: The MDE framework significantly outperformed state-of-the-art TWS and multi-criteria methods in out-of-domain scenarios.
  • Cross-lingual Performance: DSE showed significant improvements on in-domain Chinese and Urdu tasks. MDE also provided substantial gains for Chinese out-of-domain scenarios.
  • Downstream Task Impact: For tasks dependent on word segmentation quality (like NER), improvements from DSE and data augmentation were substantial, increasing F1 scores from 63.46% to 72.27%. For tasks less dependent on WS (text classification, sentiment analysis), the impact was less pronounced.

Limitations and Future Work

  • Special Character Handling: The paper notes that even their improved models struggle with special characters (e.g., '+') due to their rarity in the WS160 corpus.
  • Small Corpora for MDE: The MDE framework showed minimal improvement on Japanese datasets when corpora were very small, suggesting a minimum data threshold might be needed for effective ensembling.
  • Future Work: The authors plan to explore novel techniques such as Transformers and contrastive learning for future improvements.

Summary updated

BibTeX
@inproceedings{limkonchotiwat2021handling,
  author = {Peerat Limkonchotiwat and Wannaphong Phatthiyaphaibun and Raheem Sarwar and Ekapol Chuangsuwanich and Sarana Nutanong},
  title = {{Handling Cross- and Out-of-Domain Samples in Thai Word Segmentation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure