Detecting Parser Errors Using Web-based Semantic Filters

Alexander Yates, Stefan Schoenmackers, Oren EtzioniEMNLP 2006

Abstract

Statistical natural language parsers often produce semantically implausible results when applied to the heterogeneous and messy text found on the Web. This paper introduces WOODWARD, a system designed to validate parser outputs by converting them into logical semantic interpretations and verifying these against the vast scale of web data. By employing a suite of filters—including PMI-based, verb arity, and question-answering checks—WOODWARD identifies and discards nonsensical parses. Experimental results demonstrate that this approach can reduce error rates by up to 67% on question-answering tasks and 20% on standard benchmarks like the Penn Treebank.

Introduction

Statistical parsers are fundamental to many NLP tasks, yet they suffer from a significant "brittleness" problem. While they perform well on the specific domains they were trained on (such as the Wall Street Journal), their accuracy often plummets when faced with varied web text.

A core issue is that these parsers frequently make "wildly implausible" mistakes—for example, incorrectly attaching a prepositional phrase so that a sentence implies "operations of 69 cents" instead of "earnings of 69 cents." Humans easily catch these errors because we possess world knowledge that makes "operations of cents" semantically nonsensical. This research addresses the challenge of providing computers with a similar "sanity check" by using the Web as a massive proxy for world knowledge.

Methodology

The authors introduce Web-based semantic filtering, a process that follows a specific pipeline to identify parser errors:

  1. Semantic Interpretation: The system takes a parse tree (generated by a standard tool like the Collins parser) and converts it into a Relational Conjunction (RC)—a logic-based representation of the sentence's predicates and arguments.
A sample parse tree for a question about oil-producing states, which serves as the input for semantic interpretation.
  1. The WOODWARD Filter: The RC is then processed through four increasingly sophisticated filters to check for plausibility:
  2. PMI-Based Filter (PBF): Uses Pointwise Mutual Information based on search engine hit counts to determine if specific word pairs (e.g., a verb and its object) appear together frequently on the Web.
  3. Verb Arity Sampling Test (VAST): Targets "arity" errors where a parser assigns an incorrect number of arguments to a verb. It uses targeted web queries to see how many arguments the verb typically takes in natural usage.
  4. TextRunner Filter: Queries a database of over a billion extracted triples (subject, predicate, object) to see if the specific relationship in the parse has been previously observed.
  5. Question Answering (QA) Filter: Specifically for questions, it attempts to find a valid answer on the Web using the parse. If an answer exists, the parse is likely correct.

The system utilizes a Support Vector Machine (SVM) classifier to combine these signals and decide whether a parse should be accepted or rejected.

Experimental Setup

WOODWARD was evaluated on two primary datasets:

  • TREC 2004: A set of 55 "list" and "factoid" questions used in standard question-answering benchmarks.
  • Penn Treebank (Section 23): The "gold standard" benchmark for English parsing, consisting of Wall Street Journal articles.
  • Training Data: The filters were trained on TREC 2002/2003 questions and Section 22 of the Penn Treebank.

Key Results

WOODWARD showed significant improvements over the baseline, which assumes all parses are correct. The system proved highly effective at detecting errors that traditional syntactic models miss.

Performance comparison between the baseline and WOODWARD across TREC and WSJ datasets, highlighting significant error reduction.
  • TREC Questions: Reduced the error rate by 67%. The system was particularly effective here because the QA filter provides a very strong signal for correctness.
  • Penn Treebank: Reduced the error rate by 20%. This task is more challenging due to complex sentence structures and clause adjuncts that are intrinsically difficult to verify semantically.
  • Filter Synergy: Different filters caught different types of errors. VAST was excellent at catching verb argument errors, while PBF was better for prepositional attachments.

Discussion and Future Work

The fundamental insight of this work is that the sheer scale of the Web can overcome data sparseness in NLP. By recognizing when a resulting meaning is absurd, the system can spot "wrong" parses without necessarily knowing the "right" grammar.

However, limitations remain. The system assumes the input sentence itself makes sense; if a sentence is nonsensical, a correct parse might be rejected. Additionally, the method requires a corpus (the Web) that contains information related to the sentence being parsed. Future work may involve integrating these semantic filters directly into the parsing process to help the parser choose the correct structure from the start, rather than acting as a post-hoc judge. Furthermore, moving from live web queries to local indices like TextRunner will be essential for scaling the system's computational efficiency.

Summary updated

BibTeX
@inproceedings{yates2006detecting,
  author = {Alexander Yates and Stefan Schoenmackers and Oren Etzioni},
  title = {{Detecting Parser Errors Using Web-based Semantic Filters}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2006},
}
Figure