A Collaborative Framework for Collecting Thai Unknown Words from the Web

Choochart Haruechaiyasak, Chatchawal Sangkeettrakarn, Pornpimon Palingoon, Sarawoot Kongyoung, Chaianun DamrongratACL 2006

Overview

Thai is a "non-segmenting" language, meaning words are written continuously without spaces. For natural language processing (NLP) tasks like Machine Translation or Information Retrieval, computers must first segment text into individual words. However, "unknown words"—such as new slang, technical terms, or transliterated names—frequently break these segmenters. This paper proposes a hybrid framework that combines automated web crawling with human verification to maintain an up-to-date Thai lexicon.

The Collaborative Framework

The system consists of two primary components designed to bridge the gap between automated detection and human expertise.

1. Information Agent

The Information Agent uses a web crawler and HTML parser to harvest text from targeted online sources. This ensures the system is exposed to the most current language usage as it emerges on the web.

2. Unknown-Word Analyzer

This core engine processes the harvested text through several stages:

  • Detection: Using the LEXiTRON dictionary as a baseline, a "longest-matching" segmenter identifies sequences that do not match known words.
  • Boundary Identification: A statistical string pattern-matching algorithm determines word boundaries by analyzing the frequency of these unknown segments across different contexts.
  • Morphological Analysis: Thai-specific rules (e.g., constraints on starting characters or tonal marks) help merge fragments into coherent word candidates.
System architecture of the collaborative unknown-word collection framework.

Finally, these candidates are presented through a web interface, allowing users to verify or edit them before they are added to the master dictionary.

Key Insights

The researchers observed that unknown words follow a Zipfian distribution, where a small number of words appear very frequently while most appear rarely. Crucially, only 3% of unknown words appeared only once in their corpus, validating the use of statistical pattern matching. The study also distinguishes between "explicit" unknowns (entirely new strings like transliterations) and "hidden" unknowns (new combinations of existing words, such as names).

Evaluation and Results

The framework was tested on a corpus of over 8,000 newspaper articles. The system prioritized high recall (detection rate) to ensure as many new words as possible were flagged for human review.

Detection performance across different segment merging approaches.

As shown in the table above, the system achieved a peak detection rate of 96.6%. While automated boundary identification accuracy was lower (approx. 36%), the researchers found that merging segments (up to 5 characters) based on morphological rules significantly reduced computational overhead (by 3x) while maintaining a high detection rate. This trade-off is acceptable in a collaborative system where human editors can easily correct minor boundary errors.

Conclusion and Future Work

This framework provides a scalable solution for keeping Thai NLP tools current. Future research will focus on improving boundary accuracy and expanding the system's capability to detect "hidden" unknown words, such as complex named entities, through more advanced n-gram and phrase chunking techniques.

Summary updated

BibTeX
@inproceedings{haruechaiyasak2006collaborative,
  author = {Choochart Haruechaiyasak and Chatchawal Sangkeettrakarn and Pornpimon Palingoon and Sarawoot Kongyoung and Chaianun Damrongrat},
  title = {{A Collaborative Framework for Collecting Thai Unknown Words from the Web}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2006},
}
Figure