Abstract
Semantic parsing has transitioned from symbolic, grammar-driven systems to data-intensive neural models. While neural models achieve superior performance on aggregate metrics, their qualitative strengths and weaknesses remain poorly understood. This paper presents a new factorization-based neural parser and conducts a "phenomenon-oriented" comparison against a classic rule-based system (ACE/ERG). Our results show that while neural models excel at local lexical patterns and robustness, they struggle with global structural coherence and rare grammatical constructions compared to knowledge-intensive systems.
1. Introduction
The field of semantic parsing—the task of mapping natural language to machine-readable meaning representations—is currently divided between two philosophies. Knowledge-intensive systems, such as those based on Head-driven Phrase Structure Grammar (HPSG), rely on explicit linguistic rules and hand-coded lexicons. In contrast, data-intensive neural models learn patterns directly from large-scale corpora. While neural models produce higher overall scores, these metrics often act as "black boxes" that obscure specific linguistic failures. This study aims to move beyond aggregate scores like SMATCH to understand the specific linguistic competencies and failure modes of each approach.
2. Methodology: The Factorization-Based Neural Parser
The authors introduce a novel neural parser that treats semantic parsing as a factorization problem. The architecture consists of two main components:
- Concept Identification: Utilizing a BiLSTM with ELMo embeddings and POS tags to identify nodes within the semantic graph. The researchers found that the addition of the BiLSTM layer (the "ELMo*" model) was crucial for capturing the long-distance dependencies required for accurate semantic role labeling.
- Dependency Identification: Employing a non-linear transformation (MLP) to score potential edges between identified nodes.
A significant innovation in this model is the use of delexicalization. To combat data sparsity, specific lexical items are replaced with generalized predicates. For example, the word "introduced" might be mapped to a placeholder like *_v_to. This allows the model to learn structural patterns that generalize across different verbs and nouns, even when specific words appear infrequently in the training data.
3. Evaluation Framework
The systems were evaluated using DeepBank v1.1, which provides gold-standard Elementary Dependency Structures (EDS). To probe specific linguistic capabilities, the authors utilized the BFOZ suite—a collection of 1,000 sentences covering complex phenomena like expletives, right-node raising, and tough-adjectives—alongside a custom "phenomenon-oriented" test suite designed to evaluate lexical constructions and basic argument structures.
4. Comparative Insights: Local vs. Global Competence
The analysis revealed a fundamental trade-off between local robustness and global coherence:
- Structural "Frankensteins": Rule-based parsers like ACE are constrained by strict grammatical logic; if they produce a parse, it is almost always structurally coherent. Neural models, however, can generate "Frankenstein" graphs—locally plausible connections that are globally nonsensical because they lack an underlying formal grammar.
- Local Pattern Mastery: Neural models are "local heroes," significantly outperforming symbolic systems on Named Entity Recognition (90.55 vs 82.92). They also show remarkable data efficiency for local tasks, reaching high performance with only 10-20% of the training data.
- The Long Tail of Language: Symbolic systems maintain high precision on rare constructions, such as the "way" construction ("He dug his way out"), because they rely on well-defined rules. Neural models, dependent on statistical frequency, often fail when encountering these "long tail" phenomena due to insufficient signal in the training data.
- Grammatical Precision: The symbolic ACE parser outperformed the neural model on specific complex structures, such as it-expletives (91% vs 74% recall) and core argument roles (ARG3) in basic verbs.
5. Results and Conclusion
The proposed ELMo neural parser achieved a state-of-the-art 95.05 SMATCH score* on the EDS test set, a significant improvement over previous transition-based models. Despite this numerical success, the qualitative analysis suggests that neural models still lack the structural "common sense" inherent in grammar-driven systems.
The authors suggest that the future of semantic parsing lies in hybridization. Potential paths include using neural tagging to assist rule-based parsers with unknown words or incorporating structural inference and explicit syntactic knowledge into neural architectures to enforce global coherence. By combining the statistical robustness of data-intensive models with the structural integrity of knowledge-intensive systems, the field can move toward more reliable and interpretable semantic understanding.
Summary updated