A Comparative Analysis of Knowledge-Intensive and Data-Intensive Semantic Parsers

Junjie Cao, Zi Lin, Weiwei Sun, Xiaojun WanarXiv 2019 · arXiv

Abstract

Semantic parsing has transitioned from symbolic, grammar-driven systems to data-intensive neural models. While neural models achieve superior performance on aggregate metrics, their qualitative strengths and weaknesses remain poorly understood. This paper presents a new factorization-based neural parser and conducts a "phenomenon-oriented" comparison against a classic rule-based system (ACE/ERG). Our results show that while neural models excel at local lexical patterns and robustness, they struggle with global structural coherence and rare grammatical constructions compared to knowledge-intensive systems.

1. Introduction

The field of semantic parsing—the task of mapping natural language to machine-readable meaning representations—is currently divided between two philosophies. Knowledge-intensive systems, such as those based on Head-driven Phrase Structure Grammar (HPSG), rely on explicit linguistic rules and hand-coded lexicons. In contrast, data-intensive neural models learn patterns directly from large-scale corpora. While neural models produce higher overall scores, these metrics often act as "black boxes" that obscure specific linguistic failures. This study aims to move beyond aggregate scores like SMATCH to understand the specific linguistic competencies and failure modes of each approach.

2. Methodology: The Factorization-Based Neural Parser

The authors introduce a novel neural parser that treats semantic parsing as a factorization problem. The architecture consists of two main components:

  • Concept Identification: Utilizing a BiLSTM with ELMo embeddings and POS tags to identify nodes within the semantic graph. The researchers found that the addition of the BiLSTM layer (the "ELMo*" model) was crucial for capturing the long-distance dependencies required for accurate semantic role labeling.
  • Dependency Identification: Employing a non-linear transformation (MLP) to score potential edges between identified nodes.

A significant innovation in this model is the use of delexicalization. To combat data sparsity, specific lexical items are replaced with generalized predicates. For example, the word "introduced" might be mapped to a placeholder like *_v_to. This allows the model to learn structural patterns that generalize across different verbs and nouns, even when specific words appear infrequently in the training data.

Delexicalization mapping for the sentence 'The drug was introduced in West Germany this year'.

3. Evaluation Framework

The systems were evaluated using DeepBank v1.1, which provides gold-standard Elementary Dependency Structures (EDS). To probe specific linguistic capabilities, the authors utilized the BFOZ suite—a collection of 1,000 sentences covering complex phenomena like expletives, right-node raising, and tough-adjectives—alongside a custom "phenomenon-oriented" test suite designed to evaluate lexical constructions and basic argument structures.

Example of an Elementary Dependency Structure (EDS) graph for a passive sentence with temporal and locative modifiers.

4. Comparative Insights: Local vs. Global Competence

The analysis revealed a fundamental trade-off between local robustness and global coherence:

  • Structural "Frankensteins": Rule-based parsers like ACE are constrained by strict grammatical logic; if they produce a parse, it is almost always structurally coherent. Neural models, however, can generate "Frankenstein" graphs—locally plausible connections that are globally nonsensical because they lack an underlying formal grammar.
  • Local Pattern Mastery: Neural models are "local heroes," significantly outperforming symbolic systems on Named Entity Recognition (90.55 vs 82.92). They also show remarkable data efficiency for local tasks, reaching high performance with only 10-20% of the training data.
  • The Long Tail of Language: Symbolic systems maintain high precision on rare constructions, such as the "way" construction ("He dug his way out"), because they rely on well-defined rules. Neural models, dependent on statistical frequency, often fail when encountering these "long tail" phenomena due to insufficient signal in the training data.
  • Grammatical Precision: The symbolic ACE parser outperformed the neural model on specific complex structures, such as it-expletives (91% vs 74% recall) and core argument roles (ARG3) in basic verbs.

5. Results and Conclusion

The proposed ELMo neural parser achieved a state-of-the-art 95.05 SMATCH score* on the EDS test set, a significant improvement over previous transition-based models. Despite this numerical success, the qualitative analysis suggests that neural models still lack the structural "common sense" inherent in grammar-driven systems.

The authors suggest that the future of semantic parsing lies in hybridization. Potential paths include using neural tagging to assist rule-based parsers with unknown words or incorporating structural inference and explicit syntactic knowledge into neural architectures to enforce global coherence. By combining the statistical robustness of data-intensive models with the structural integrity of knowledge-intensive systems, the field can move toward more reliable and interpretable semantic understanding.

Summary updated

Abstract

We present a phenomenon-oriented comparative analysis of the two dominant approaches in task-independent semantic parsing: classic, knowledge-intensive and neural, data-intensive models. To reflect state-of-the-art neural NLP technologies, we introduce a new target structure-centric parser that can produce semantic graphs much more accurately than previous data-driven parsers. We then show that, in spite of comparable performance overall, knowledge- and data-intensive models produce different types of errors, in a way that can be explained by their theoretical properties. This analysis leads to new directions for parser development.
BibTeX
@misc{cao2019comparative,
  author = {Junjie Cao and Zi Lin and Weiwei Sun and Xiaojun Wan},
  title = {{A Comparative Analysis of Knowledge-Intensive and Data-Intensive Semantic Parsers}},
  year = {2019},
  eprint = {1907.02298},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.02298},
}
Figure