TL;DR
This paper introduces a novel morphological analysis system that significantly expands lexical coverage for natural language processing by using a relatively small core lexicon augmented with ~1200 knowledge-based rules. This approach achieves robust coverage for unrestricted domains, outperforming much larger lexicons and demonstrating high accuracy in syntactic category assignment and root identification.
What Problem Does This Paper Address?
Natural language processing (NLP) applications often require a vast vocabulary to handle diverse and unrestricted text. While large lexicons exist, they inevitably fail to cover all encountered words, especially in specialized domains or with novel terminology. This paper tackles the challenge of achieving robust lexical coverage by developing a sophisticated morphological analysis system that can intelligently infer the meaning and grammatical role of unknown words by understanding their structure (prefixes, suffixes, compounds). This is crucial for applications that need to process text without prior knowledge of every word.
Methods
The core of the paper is the description of an "aggressive" morphological analysis system. This system integrates with an extensive lexicon, a semantic ontology, and a syntactic analysis system. Its key features include:
- Knowledge-Based Morphology: Instead of purely statistical methods, it uses approximately 1200 hand-crafted, knowledge-based morphological rules.
- Integrated System: The morphological analyzer doesn't operate in isolation; it consults and augments the lexicon, semantic ontology, and syntactic parser. For instance, semantic relationships help disambiguate word meanings (e.g., "cupful" as a unit of measure vs. "hopeful" as an adjective).
- Ordered Preferential Rules: The system employs a set of rules applied in a specific, preferential order. This hierarchy prioritizes more reliable analyses (e.g., known roots with suffixes) over less certain ones, aiming to produce a small number (1-3) of correct interpretations.
- Heuristic Approach: The rules are heuristic, meaning they are designed to make plausible inferences in the face of uncertainty, rather than guaranteeing perfect accuracy. The goal is to derive a usable set of part-of-speech classifications and identify root-plus-affix or compound structures.
- Special Form Handling: The system includes specific tests for recognizing and handling various "special forms" like numbers, Roman numerals, ordinals, alphanumeric sequences, and even email addresses and URLs.
- Recursive Application: The morphological analysis can be applied recursively to hypothesized roots, allowing for the analysis of complex derived words. Caching is used to prevent infinite loops and manage computational cost.
Datasets
The paper evaluates its system using:
- Brown Corpus: A well-known corpus of American English text.
- A random sample of 100 word tokens.
- A random sample of 100 distinct word types.
- Technical Literature Corpus: A 3.1-million-word corpus of technical literature was used to demonstrate the need for robust coverage, showing that a significant percentage of words were not covered by a large hypothetical lexicon.
The paper does not introduce new datasets but uses existing ones for evaluation.
Key Insights and Ideas
- Morphology as a Powerful Lexicon Extender: The central idea is that a well-designed morphological system can dramatically increase lexical coverage without needing an astronomically large lexicon. This is particularly impactful for domains with specialized or evolving vocabulary.
- Integration is Key: The success of the morphological analysis is heavily reliant on its integration with semantic and syntactic information. This allows for disambiguation and more accurate interpretation of unknown words.
- Ordered Rules for Robustness: The preferential ordering of rules is crucial. It ensures that more reliable analyses are attempted first, preventing less confident suffix-based analyses from overriding potentially correct prefix or compound interpretations.
- Heuristics for Real-World Uncertainty: The heuristic nature of the rules is a pragmatic choice. It acknowledges that perfect analysis of every unknown word is impossible and focuses on providing useful, plausible interpretations to enable parsing.
- "Known" Roots as a Stepping Stone: The concept of a "known" root (which can include derived forms of known words) is a clever way to bootstrap the analysis process and increase the reliability of suffix rule applications.
Key Results
- Superior Lexical Coverage: The system, using ~1200 rules and a 39,000-word lexicon, achieved lexical coverage exceeding that of an 80,000-base-form or 150,000-word-form English lexicon.
- High Accuracy on Unknown Words: When applied to 72 unknown word types from the Brown corpus:
- 97% of syntactic category assignments received a grade of B or better (with 86% being A).
- 95% of root words were identified correctly.
- 100% of count/mass distinctions were made correctly for nouns.
- Efficiency: Morphological analysis was found to be a negligible part of the overall text-processing time, indicating its efficiency.
Limitations and Future Work
- Subjectivity in Rule Creation: The ~1200 rules are knowledge-based, implying a significant manual effort and potential for subjectivity in their design and ordering.
- Heuristic Nature: While a strength, the heuristic approach means that incorrect analyses are possible, especially for complex or ambiguous words.
- Future Work: The paper implicitly suggests that refining the rule set, expanding the semantic ontology, and further integrating with syntactic parsers are ongoing areas. The success on technical literature also points to its applicability in specialized domains, which could be a direction for future exploration. The paper also mentions that the "guess" rule for nouns and verbs can be adjusted based on the comprehensiveness of the core lexicon, hinting at adaptive strategies.
Summary updated