Overview
Thai is a "non-segmenting" language, meaning words are written continuously without spaces. For natural language processing (NLP) tasks like Machine Translation or Information Retrieval, computers must first segment text into individual words. However, "unknown words"—such as new slang, technical terms, or transliterated names—frequently break these segmenters. This paper proposes a hybrid framework that combines automated web crawling with human verification to maintain an up-to-date Thai lexicon.
The Collaborative Framework
The system consists of two primary components designed to bridge the gap between automated detection and human expertise.
1. Information Agent
The Information Agent uses a web crawler and HTML parser to harvest text from targeted online sources. This ensures the system is exposed to the most current language usage as it emerges on the web.
2. Unknown-Word Analyzer
This core engine processes the harvested text through several stages:
- Detection: Using the LEXiTRON dictionary as a baseline, a "longest-matching" segmenter identifies sequences that do not match known words.
- Boundary Identification: A statistical string pattern-matching algorithm determines word boundaries by analyzing the frequency of these unknown segments across different contexts.
- Morphological Analysis: Thai-specific rules (e.g., constraints on starting characters or tonal marks) help merge fragments into coherent word candidates.
Finally, these candidates are presented through a web interface, allowing users to verify or edit them before they are added to the master dictionary.
Key Insights
The researchers observed that unknown words follow a Zipfian distribution, where a small number of words appear very frequently while most appear rarely. Crucially, only 3% of unknown words appeared only once in their corpus, validating the use of statistical pattern matching. The study also distinguishes between "explicit" unknowns (entirely new strings like transliterations) and "hidden" unknowns (new combinations of existing words, such as names).
Evaluation and Results
The framework was tested on a corpus of over 8,000 newspaper articles. The system prioritized high recall (detection rate) to ensure as many new words as possible were flagged for human review.
As shown in the table above, the system achieved a peak detection rate of 96.6%. While automated boundary identification accuracy was lower (approx. 36%), the researchers found that merging segments (up to 5 characters) based on morphological rules significantly reduced computational overhead (by 3x) while maintaining a high detection rate. This trade-off is acceptable in a collaborative system where human editors can easily correct minor boundary errors.
Conclusion and Future Work
This framework provides a scalable solution for keeping Thai NLP tools current. Future research will focus on improving boundary accuracy and expanding the system's capability to detect "hidden" unknown words, such as complex named entities, through more advanced n-gram and phrase chunking techniques.
Summary updated