Integrating Word Boundary Identification with Sentence Understanding

Kok-Wee GanACL 1993 · DOI

Introduction

In Chinese Natural Language Processing (NLP), the traditional "segment-then-parse" pipeline presents a significant bottleneck. Unlike English, written Chinese lacks explicit word boundaries (spaces). Standard systems typically employ a standalone "word segmentor" to tokenize character strings before any semantic or syntactic analysis occurs.

This sequential approach is fundamentally flawed because many character sequences are inherently ambiguous. For instance, the characters for "person" (ren) and "birth" (sheng) might correctly be separate words in one context, yet a statistical segmenter might erroneously group them into the single word "life" (rensheng) in another. By the time a traditional system identifies a semantic inconsistency, the incorrect segmentation is already finalized. This paper argues that word boundary identification is inseparable from sentence understanding; semantic context is often required to determine where words end.

Methodology: The Crystallization Model

The proposed model is inspired by Douglas Hofstadter’s Copycat architecture, treating sentence understanding as a "crystallization" process. In this metaphor, characters (ions) gradually form into words and phrases (crystals) as the system "cools."

The architecture relies on several key components:

  • Codelets: These are small, independent agents running in parallel. "Bottom-up" codelets identify statistical associations between characters, while "top-down" codelets search for specific syntactic or semantic structures.
  • The Coderack: A probabilistic queue where codelets await execution.
  • Temperature: A global variable representing the system's "unhappiness" or uncertainty. Semantic violations (e.g., an aspect marker attached to a noun) cause the temperature to rise, increasing randomness and allowing the system to "dissolve" existing word groupings to explore alternatives.
  • Parallel Terraced Scan: The system explores multiple interpretations simultaneously, allocating more resources to promising paths while maintaining less likely alternatives.

Data and Statistical Foundation

The model's initial "bottom-up" guesses are driven by statistical data from a corpus provided by Liang Nanyuan (Beijing University of Aeronautics and Astronautics). This corpus includes 46,520 words with a total usage frequency exceeding 13 million. These statistics allow the system to calculate "mutual information" between characters, providing a probabilistic starting point for segmentation.

Core Innovations

The integration of segmentation and parsing yields several conceptual shifts:

  • Segmentation as a Side Effect: Word boundaries are not the primary goal but a byproduct of constructing a coherent semantic network. Groupings that impede understanding are discarded.
  • Dynamic Fluidity via "Temperature": The temperature variable allows the system to lock onto easy associations while remaining volatile enough to pivot when encountering logical dead-ends.
  • Reduced Redundancy: Merging stages avoids duplicating morphological and syntactic checks.
  • Semantic Constraints as Error Detectors: Selectional restrictions (e.g., the verb kaifa [develop] requiring a location or material object) prove more effective at resolving ambiguities than simple dictionary matching.

Experimental Analysis

The paper evaluates the model using qualitative results from three "problem sentences" that typically cause statistical segmentors to fail.

  1. Sentence (1): The system corrects the false grouping of rensheng ("life") by recognizing that a subsequent aspect marker le requires a preceding verb. This forces a re-segmentation into benren ("alone") and sheng ("give birth").
  2. Sentence (2): It resolves the ambiguity of shifen (which can mean "ten marks" or the adverb "very") by identifying that "very" is syntactically invalid at the end of the specific sentence structure.
  3. Sentence (3): The model correctly identifies boundaries in a complex phrase regarding oil resources.
Example of complex word boundary identification in Chinese sentence (3).

In Sentence (3), as shown above, the system applies thematic role interpretations to rule out the word heshang ("monk"), which a naive segmenter might find by grouping the characters for "and" (he) and "yet" (shang). By understanding the broader context of "developed and not yet developed oil resources," the system correctly partitions the characters.

Limitations and Future Work

While the framework is theoretically robust, the author notes several limitations:

  • Scale: The current evaluation focuses on "hand-worked" test cases rather than large-scale automated benchmarks.
  • Rigidity of Constraints: The principle that every constituent must be meaningfully related is currently "too tight," making it difficult to process ill-formed sentences or metonymy.
  • Conceptual Network Detail: The specific structure of the linguistic conceptual network is only briefly touched upon, with further details deferred to other technical reports.

Summary updated

BibTeX
@inproceedings{gan1993integrating,
  author = {Kok-Wee Gan},
  title = {{Integrating Word Boundary Identification with Sentence Understanding}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {1993},
  doi = {10.3115/981574.981621},
}
Figure