TL;DR
This paper tackles the challenge of building accurate Natural Language Processing (NLP) systems without requiring deep linguistic expertise from developers. They propose a novel approach that combines a general-purpose, broad-coverage parser (Minipar) with a domain-specific, shallow parser to achieve both wide applicability and high accuracy for specific tasks. This hybrid system significantly improves parsing accuracy compared to using a single parser alone.
What Problem Does This Paper Address?
The rapid growth of NLP applications, especially those involving speech input, has created a demand for accessible NLP toolkits. However, most programmers lack the linguistic knowledge to develop these systems from scratch. Existing broad-coverage parsers, while general, often lack the accuracy needed for specific commercial domains. Conversely, domain-specific parsers are accurate but not general. This paper addresses the gap by proposing a method to equip a general NLP toolkit with a parser that is both broadly applicable and highly accurate for specific domains, enabling non-experts to build robust NLP interfaces.
Methods
The core of the paper's contribution lies in a hybrid parsing strategy. They leverage Minipar, a well-established broad-coverage parser, for its generality. To address Minipar's limitations in domain-specific accuracy, they introduce and integrate a domain-adaptable shallow parser. This shallow parser focuses on extracting key verb-argument structures and local dependencies, which are crucial for mapping natural language input to commands.
The integration process involves:
- Running both parsers in parallel: For any given input sentence, both Minipar and the shallow parser generate their respective parse trees.
- An "error detector" module: This module uses a set of heuristics to identify potentially incorrect parse trees from each parser.
- A decision-making module: Based on the error detector's output and a set of predefined heuristics, the system selects the "better" parse tree. These heuristics prioritize the shallow parser for specific cases where it excels (e.g., tagging, PP attachment) and Minipar for others (e.g., negative/passive sentences, sentences with the verb "be").
Datasets
The authors developed their system using a corpus of email messages from their customers, which contained questions, comments, and inquiries related to their document-conversion products. This corpus was preprocessed to remove extraneous information. A test corpus of 1000 inquiries was created from this data to evaluate the parsers. They also mention using the SUSANNE corpus for evaluating Minipar's general performance.
Key Insights and Ideas
- Hybrid Parsing is Key: The most significant insight is that combining a general parser with a domain-specific shallow parser offers a powerful solution to the generality-accuracy trade-off in NLP.
- Domain Adaptation is Crucial: Even broad-coverage parsers struggle with domain-specific nuances. The shallow parser's ability to be adapted to specific phenomena (like verb subcategorization and PP attachment) is vital for achieving high accuracy.
- Heuristics for Tree Selection: Developing effective heuristics to choose between two parse trees is critical. The paper demonstrates that a rule-based approach, informed by the known strengths and weaknesses of each parser, can significantly improve overall accuracy.
- The Importance of Local Dependencies: For many NLP tasks, especially command generation, local dependency information (verb-argument structures) is more critical than a full, deep parse. The shallow parser's focus on this is a key advantage.
Key Results
The paper reports significant improvements in parsing accuracy:
- Minipar alone achieved an accuracy of 77.6% on their test corpus.
- The extended parser (combining Minipar and the shallow parser with heuristics) achieved an accuracy of 82.5%. This represents a 7% improvement over using Minipar alone.
- The extended parser was able to choose the correct parse tree 88% of the time, with Minipar trees being chosen as the final output in most cases (88%).
- The computational expense of the extended parser was about 2.5 times slower than Minipar alone, but they note this is due to the shallow parser not being optimized.
Limitations and Future Work
- Computational Cost: The current implementation of the extended parser is slower than using Minipar alone due to the parallel processing and the unoptimized shallow parser.
- Heuristic Limitations: The error detector's accuracy is not perfect (75% for Minipar, 83% for shallow parser), meaning some incorrect trees might be selected. Tagging errors and some attachment errors are particularly difficult to detect.
- Future Work: The authors plan to improve the efficiency and accuracy of the shallow parser. They also intend to test the parser on a different domain to assess the effort required for domain switching and further validate their approach.
Summary updated