Automatic Semantic Sequence Extraction from Unrestricted Non-Tagged Texts

Shiho Nobesawa, Hiroaki Saito, Masakazu NakanishiCOLING 2000

TL;DR

This paper introduces a novel, language-independent method for automatically extracting meaningful "linky sequences" from untagged text, relying solely on statistical co-occurrence information between letters. This approach effectively addresses the limitations of dictionary-based NLP tools when dealing with casual language and domain-specific terms, demonstrating significant success in recovering sequences missed by traditional parsers.

What Problem Does This Paper Address?

Natural Language Processing (NLP) tools often struggle with unrestricted, non-tagged texts, particularly those written in casual language or containing domain-specific jargon. Traditional methods, like morphological processing and syntactic parsing, heavily rely on dictionaries and predefined rules. This reliance creates a significant bottleneck: when encountering words or phrases not present in their dictionaries (unknown words), these systems often fail, leading to parsing errors. This is especially problematic in domains like email communication, where informal language, slang, and unique expressions are common. The paper aims to bridge this gap by proposing a method that can extract meaningful sequences without requiring extensive lexicons or domain-specific preparation, making NLP more robust for real-world, uncurated text.

Methods

The core of the proposed system, termed LSE (Linky Sequence Extraction), is a statistical approach that estimates the likelihood of letter sequences being meaningful units. It operates in two main phases:

  • Training: The system learns statistical information from a large corpus of untagged text. Specifically, it calculates "linking scores" between neighboring letters. This is achieved using a concept called "d-bigrams," which considers co-occurrence information not just between adjacent letters but also between letters separated by a certain distance (up to d_max). The mutual information between letter pairs at various distances is calculated to quantify their connection strength.
  • Extraction: For a given input sentence, the system calculates the linking score for all adjacent letter pairs based on the learned statistical data. These scores are then visualized as a "score graph." The system identifies "mountain-shaped" segments in this graph, representing sequences of letters with strong statistical connections. These segments are considered "linky sequences" and are extracted as meaningful units. The extraction process involves a threshold to determine which sequences are significant enough to be considered.

Crucially, this method is language-independent, requiring only a sufficiently large training corpus in the target language. It does not rely on any pre-existing lexicons or morphological parsers during the extraction phase.

Datasets

  • Training Corpus: A set of email messages sent between young female friends during 1998-1999. This corpus contained 351 email messages, 7,865 sentences, and approximately 176,380 letters.
  • Test Corpus: A separate set of email messages sent between young female friends during 1999. This corpus comprised 1,118 sentences and 24,160 letters.
  • Newspaper Corpus: Used as a "base domain" for comparison in some experiments, though email was the primary focus.
  • Postcard Corpus: Used as a "target domain" for experiments illustrating the system's ability to adapt to specific domains.

Key Insights and Ideas

  • Statistical Power of Letter Co-occurrence: The paper strongly demonstrates that simple statistical information derived from letter co-occurrences alone can be surprisingly effective in identifying meaningful linguistic units, even without explicit word boundaries or semantic knowledge.
  • Language Independence: The method's reliance on statistical patterns rather than language-specific rules makes it a powerful tool for processing diverse languages, especially those with complex morphology or no explicit word delimiters (like Japanese).
  • Robustness to Casual Language: The system's ability to extract sequences missed by dictionary-based parsers highlights its strength in handling informal language, slang, and variations in expression common in everyday communication like emails.
  • "Linky Sequences" as Meaningful Units: The concept of "linky sequences" as statistically connected letter groups offers a novel perspective on segmenting text, moving beyond traditional word-based units to capture potentially meaningful sub-word or phrase-level units.

Key Results

The system achieved promising results in extracting semantic sequences from Japanese email texts:

  • Recovery of Undefined Words: The LSE system successfully recovered 69.06% of sequences that were tagged as "undefined words" by the dictionary-based Japanese morphological parser, ChaSen.
  • High Precision for Frequent Sequences: For sequences appearing more than 10 times in the test corpus, the system achieved an extraction accuracy of 81.85%.
  • Handling of Representation Changes: The system was effective in salvaging semantically meaningful sequences that were not in dictionaries due to various representation changes (e.g., phonetic spellings, emphasis, slang), recovering 56.78% of undefined words related to these changes.
  • Segmentation Failure Recovery: The system also showed effectiveness in recovering sequences that ChaSen failed to segment correctly, achieving 70.88% success in these cases.

Limitations and Future Work

  • Limitations:
    • The system's performance is dependent on the size and quality of the training corpus.
  • While effective, the system can still over-segment sequences, particularly when dealing with counting suffixes in Japanese, due to the lack of explicit word boundary information.
  • The paper acknowledges that for some undefined words, ChaSen could not segment them correctly even when they were neighboring, and the LSE system's performance in these specific cases is not fully detailed.
  • Future Work:
    • Combining with Dictionaries: Exploring ways to integrate the extracted sequences with existing dictionaries to enhance precision and recall.
  • POS Tagging: Investigating the use of Part-of-Speech (POS) tags for the extracted sequences to facilitate their use with other NLP tools.
  • Language Extension: Further testing and application of the language-independent system to other languages, such as English, for tasks like proper noun extraction.

Summary updated

BibTeX
@inproceedings{nobesawa2000automatic,
  author = {Shiho Nobesawa and Hiroaki Saito and Masakazu Nakanishi},
  title = {{Automatic Semantic Sequence Extraction from Unrestricted Non-Tagged Texts}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2000},
}
Figure