TL;DR
This paper introduces a method for dynamically updating a Chinese NLP system's lexicon during sentence analysis, reducing the need for manual dictionary editing. By intelligently guessing new words and their grammatical properties from context and then filtering these guesses, the system significantly improves parsing accuracy and coverage, especially for domain-specific texts.
What Problem Does This Paper Address?
Natural Language Processing (NLP) systems heavily rely on comprehensive dictionaries (lexicons) to understand language. However, dictionaries are never truly complete. New words are constantly coined, existing words change their usage, and a dictionary that's complete for one domain might be woefully inadequate for another. Traditionally, updating these dictionaries involves expensive and time-consuming manual editing or batch processing of external data. This approach is a never-ending task because language is dynamic. The paper addresses the challenge of keeping NLP lexicons up-to-date and robust without constant human intervention, particularly for languages like Chinese where word segmentation and lexical ambiguity can be complex.
Methods
The core of the paper's contribution is a two-stage process for dynamic lexical acquisition:
- Intelligent Guessing (Proposing Words and Attributes):
- The system uses pre-defined "contextual templates" (heuristics based on linguistic intuition) to identify potential new words or new grammatical properties (like part-of-speech or sub-categorization frames) for existing words.
- New Words: This includes single-character bound morphemes used independently (e.g., 侃 used as a verb) and new combinations of characters forming words (e.g., 接驳 for "dock").
- New Attributes: This involves identifying when a word is used with a grammatical role not listed in the existing dictionary (e.g., 同步 used as a transitive verb instead of intransitive, or 群集 used as a verb instead of a noun).
- These guesses are intentionally "over-guesses" to maximize recall.
- Filtering and Lexicalization (Harvesting New Words and Attributes):
- Competition: Proposed new words/attributes are assigned lower scores in the parsing chart. They only get selected if they are essential for achieving a valid parse. This "competition" allows the system to ignore incorrect guesses if a sentence can be parsed without them.
- Keeping Winners: If a proposed word or attribute is crucial for a successful parse, it's considered a "winner." This information is stored in an auxiliary lexicon, along with a frequency count of how often it was successfully used.
- Use the Fittest: Auxiliary lexicons are used in subsequent sentence analysis. Entries can be merged with the main lexicon, either temporarily or permanently, based on frequency thresholds. This allows the system to learn from its own parsing experiences.
Datasets
The evaluation was performed on a corpus of 121,863 sentences from Microsoft technical manuals. The authors chose this dataset because it's representative of domain-specific text with many unlisted words and novel usages, likely due to being translated from English.
Key Insights and Ideas
- Language is Dynamic, Lexicons Should Be Too: The fundamental idea is that static dictionaries are insufficient for robust NLP. Systems need to adapt to evolving language.
- Context is King for Lexical Discovery: The system leverages sentence context to infer missing lexical information, mimicking how human lexicographers work.
- Over-Guessing with Smart Filtering Works: Intentionally making many guesses (high recall) and then using the parsing process itself to filter out incorrect ones (precision) is an effective strategy.
- Auxiliary Lexicons Bridge the Gap: Storing learned information in separate, domain-specific auxiliary lexicons allows for flexible updates and the creation of specialized dictionaries.
- Domain Adaptation is a Natural Outcome: By building auxiliary lexicons from domain-specific corpora, the system can automatically create domain-specific dictionaries.
Key Results
The paper reports significant improvements in parsing performance:
- Online Guessing Only: 10.71% of sentences showed improved parses when dynamic guessing was enabled, with 1,071 net improvements out of 10,000 randomly selected sentences.
- Additional Gain with Lexicalization: When the learned information from auxiliary lexicons was used, the recognition rate of new words increased by 8.4%, potentially saving 369 additional sentences in parsing.
- Improvements were attributed 48% to new word recognition and 52% to new grammatical attributes.
- The system successfully identified 9226 unique new words in the corpus, indicating a significant portion of the vocabulary was missing from the original dictionary.
Limitations and Future Work
- Hand-Coded Templates: The current contextual templates are hand-coded heuristics based on linguists' intuition. The authors plan to use machine learning techniques to acquire these templates automatically.
- Evaluation Complexity: Evaluating the effect on grammatical attributes was noted as more complex and time-consuming than for new words.
- Over-Guessing Failures: Some sentences became worse due to "over-guessing" or failures in the filtering process, particularly when the guessing algorithm required significant processing resources.
- Future Work: The authors suggest exploring machine learning for template acquisition and further refining the filtering mechanisms.
Summary updated