MARS: A Specialized RTE System for Parser Evaluation

Rui Wang, Yi ZhangSemEval @ ACL 2010

Abstract

MARS is a specialized Recognizing Textual Entailment (RTE) system designed for the Parser Evaluation using Textual Entailment (PETE) task. By integrating syntactic dependency parsing, semantic role labeling (SRL), and deep grammatical filtering via HPSG, MARS bridges the gap between structural analysis and semantic reasoning. The system achieved a top-three finish in the SemEval-2010 PETE competition, demonstrating the efficacy of joint representations and grammaticality checks in parser evaluation.

Introduction

Traditionally, natural language parsers are evaluated by comparing their output to a "gold standard" human-annotated treebank. However, this metric does not always correlate with a parser's utility in real-world applications. The Parser Evaluation using Textual Entailment (PETE) task proposes an alternative: using a parser's output to solve a reasoning problem. If a parser correctly identifies the relationships in a complex sentence (the "Text"), a downstream system should be able to confirm if a simpler "Hypothesis" logically follows. MARS was developed to address the challenges in this "no-man's-land" between syntax (structure) and semantics (meaning).

System Architecture

The MARS system utilizes a multi-stage pipeline to transform raw text into an entailment decision.

The MARS system architecture, illustrating the pipeline from preprocessing to classification.
  1. Preprocessing & Multi-Parsing: The system employs two distinct dependency parsers—MSTParser and MaltParser—to capture syntactic structures. Simultaneously, a Semantic Role Labeler (SRL) identifies "who did what to whom" by assigning roles such as Agent and Patient.
  2. Deep Grammar Filtering: A unique component of MARS is the use of a hand-crafted Head-driven Phrase Structure Grammar (HPSG) parser. In the PETE task, hypotheses are often machine-generated and potentially ungrammatical. The HPSG parser acts as a "sanity check"; if it rejects a hypothesis, it provides a strong signal that the entailment may be false or the hypothesis is malformed.
  3. Graph-based Path Extraction: Sentences are represented as graphs where words are nodes and dependencies are edges. Using Dijkstra’s shortest path algorithm, the system identifies how key word pairs in the Hypothesis are connected within the original Text.
  4. Feature-based Classification: An SVM classifier evaluates the extracted paths based on features like directionality (ensuring the "biter" and "bitten" roles are not swapped), dependency label similarity, and whether the connection is syntactic, semantic, or joint.

Experimental Results

The system was evaluated using the SemEval-2010 Task #12 (PETE) benchmark. Training involved the Penn Treebank for POS tagging and the CoNLL 2008 Shared Task dataset for dependency and semantic role labeling.

Accuracy results for MARS using different parsers and configurations, highlighting the impact of the Grammaticality Check (+GC).

The best configuration of MARS achieved an accuracy of 66.78%, ranking 3rd overall in the competition. A key insight was the power of "Joint" representations; combining syntactic trees and semantic graphs into a single Joint Dependency Graph yielded the best results. Furthermore, the inclusion of the HPSG-based grammaticality check (+GC) consistently improved accuracy across almost all test settings, proving that deep linguistic knowledge can effectively complement data-driven statistical methods.

Conclusion

MARS demonstrates that specialized RTE modules can effectively evaluate parser performance by testing their ability to support logical reasoning. The system's success highlights the importance of directionality in dependency paths and the utility of hand-crafted grammars as noise filters. Future work will focus on handling complex linguistic phenomena such as negation and quantifiers to further refine the entailment logic.

Summary updated

BibTeX
@inproceedings{wang2010mars,
  author = {Rui Wang and Yi Zhang},
  title = {{MARS: A Specialized RTE System for Parser Evaluation}},
  booktitle = {SemEval @ ACL},
  year = {2010},
}
Figure