Automatic Corpus-Based Thai Word Extraction with the C4.5 Learning Algorithm

Virach Sornlertlamvanich, Tanapong Potipiti, Thatsanee CharoenpornCOLING 2000

TL;DR

This paper tackles the challenge of automatically extracting words from Thai, a language that lacks explicit word boundaries (spaces). They propose a novel approach using the C4.5 decision tree algorithm, trained on various linguistic features, to distinguish between actual words and non-word strings. This method achieves a respectable 85% precision, offering a promising alternative to manual dictionary creation for Thai and similar languages.

What Problem Does This Paper Address?

Many languages, like English, use spaces to clearly delineate words. However, languages such as Thai, Japanese, and others in Asia present a significant challenge for natural language processing (NLP) because they do not use spaces to separate words. This "inexplicit word boundary" makes fundamental NLP tasks like word segmentation, information retrieval, and machine translation incredibly difficult. Traditional methods often rely on manually created dictionaries, which are labor-intensive, prone to errors, and struggle to keep up with the ever-evolving nature of language, especially new words and compound terms. This paper aims to automate the process of identifying valid words within Thai text, thereby overcoming this linguistic barrier.

Methods

The core of this paper's approach is the application of the C4.5 decision tree learning algorithm. Instead of relying on pre-defined rules or extensive dictionaries, they frame the problem as a word/non-word string disambiguation task. To achieve this, they engineer a set of linguistic attributes that the C4.5 algorithm uses to build a decision tree:

  • Mutual Information (Left and Right): Measures how much two adjacent character sequences co-occur more than expected by chance. High mutual information suggests a strong linguistic connection, potentially indicating a word.
  • Entropy (Left and Right): Quantifies the disorder or randomness of characters preceding or following a given string. High entropy suggests variety, which is characteristic of words.
  • Frequency: The occurrence count of a string within the corpus, normalized by corpus size and average word length. More frequent strings are more likely to be words.
  • Length: Shorter strings are more likely to occur by chance than longer ones.
  • Functional Words: A binary attribute indicating if a string contains common Thai functional words (like "will" or "then") that might otherwise create misleading patterns.
  • First Two and Last Two Characters: Attributes that check if the beginning and end of a string conform to Thai spelling rules, using a dictionary as a reference.

The C4.5 algorithm learns from these attributes to construct a decision tree that can then classify unseen strings as either words or non-words.

Datasets

The authors utilized two main corpora for their experiments:

  • Training Corpus: A 1-MB corpus consisting of 75 articles from various fields. From this, they extracted candidate strings (2-30 characters long, occurring more than twice) and manually tagged them as words or non-words to create the training set for the C4.5 algorithm.
  • Test Corpus: Another plain 1-MB corpus, comprising 72 articles from diverse fields. This corpus was processed identically to the training corpus to extract and filter candidate strings, which were then manually tagged for evaluation.

The Thai Royal Institute dictionary was also used as a reference for the "First Two and Last Two Characters" attribute.

Key Insights and Ideas

  • Leveraging Machine Learning for Linguistic Ambiguity: The paper demonstrates that a supervised learning algorithm like C4.5 can effectively learn to distinguish words in a language without explicit boundaries, moving beyond purely rule-based or dictionary-dependent methods.
  • The Power of Combined Features: No single linguistic feature is perfect. The combination of mutual information, entropy, frequency, length, and spelling rules provides a robust set of indicators for wordhood.
  • Potential for New Word Discovery: The results suggest that the algorithm can identify words not present in existing dictionaries, highlighting its utility for expanding linguistic resources.
  • Robustness to Unseen Data: The close performance between training and test sets indicates that the learned decision tree is generalizable and not overfitted to the training data.

Key Results

The experimental results show promising performance:

  • Precision: The algorithm achieved approximately 87.3% precision on the training set and 84.1% precision on the test set. This means that out of all the strings classified as words, a high percentage were indeed actual words.
  • Recall: The recall of word extraction was 56% for both training and test sets. This indicates that the algorithm successfully identified 56% of all actual words present in the corpora.
  • Comparison to Dictionary: The recall is comparable to using the Thai Royal Institute dictionary, and the authors suggest their approach would outperform a dictionary on larger corpora.
  • New Word Identification: Over 30% of the words extracted by the decision tree were not found in the Thai Royal Institute dictionary, suggesting the algorithm's capability to discover new or evolving vocabulary.

Limitations and Future Work

  • Recall is Moderate: While precision is high, the recall of 56% indicates that a significant portion of actual words were missed. This is acknowledged as an area for improvement.
  • Corpus Size Dependency: The authors note that accuracy is expected to increase with larger corpora, implying that the current results might be limited by the size of the training and test sets.
  • Future Work: The authors plan to apply this algorithm with larger corpora to build a more comprehensive corpus-based Thai dictionary. They also express hope that this approach will be successful for other non-word-boundary languages.

Summary updated

BibTeX
@inproceedings{sornlertlamvanich2000automatic,
  author = {Virach Sornlertlamvanich and Tanapong Potipiti and Thatsanee Charoenporn},
  title = {{Automatic Corpus-Based Thai Word Extraction with the C4.5 Learning Algorithm}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2000},
}
Figure