Abstract
This volume provides linguistic, methodological, and technical background to the target representations, mode of operation, and participating systems in a 'system bake-off' for data-driven parsing into graph-structured representations of sentence meaning.
Introduction: Unifying the Tower of Babel
For decades, the field of Natural Language Processing (NLP) has faced a "balkanization" of semantic representations. Different research communities developed distinct frameworks to capture sentence meaning, ranging from logic-heavy English Resource Semantics to concept-focused Abstract Meaning Representation (AMR). Because these frameworks utilized different formalisms (trees vs. graphs), anchoring methods, and evaluation metrics, cross-framework comparison was historically impossible.
The 2019 Shared Task on Cross-Framework Meaning Representation Parsing (MRP) addressed this gap. By establishing a unified pipeline and evaluation metric, the task invited 18 teams to compete in parsing five distinct semantic frameworks, marking a landmark effort in semantic interoperability.
The MRP Framework: A Unified Formal Model
The core innovation of the 2019 task was a unifying formal model that treats all meaning representations as directed graphs. To accommodate the diversity of existing frameworks, the organizers categorized them into three "flavors" based on their relationship to the surface text:
- Flavor 0 (Bi-lexical Dependencies): Nodes correspond exactly to surface tokens (e.g., DM, PSD).
- Flavor 1 (Anchored Graphs): Nodes correspond to arbitrary spans, and multiple nodes can overlap or share anchors (e.g., EDS, UCCA).
- Flavor 2 (Unanchored Graphs): Nodes represent abstract concepts not directly tied to specific words in the sentence (e.g., AMR).
All frameworks were converted into a uniform JSON-based interchange format, allowing for a single, consistent evaluation metric based on "atomic tuples" (nodes, edges, and properties).
Architectural Trends: BERT and Parsing Strategies
Participating systems generally employed one of three architectural strategies:
- Transition-Based: Parsers that construct graphs through a sequence of actions (e.g.,
SHIFT,REDUCE,ADD_EDGE), adapted to handle cycles and reentrancy. - Factorization-Based: Systems that score potential edges between all node pairs and identify the "Maximum Spanning Connected Subgraph."
- Composition-Based: Parsers utilizing formal grammars, such as Hyperedge Replacement Grammar, to build meaning compositionally.
A dominant technical trend was the integration of BERT (Bidirectional Encoder Representations from Transformers). Teams transitioning from static embeddings to BERT reported significant F1 score improvements (often 5-10 points), confirming that deep contextual features are vital for resolving semantic ambiguity.
Experimental Setup and Datasets
The task utilized five established "graphbanks": DM, PSD, EDS, UCCA, and AMR. To test generalization, the organizers included out-of-domain data from the Brown Corpus and a newly annotated set of 100 sentences from the novel The Little Prince. This diverse data suite challenged systems to perform beyond their training distributions.
Key Insights and Results
The top-performing system, HIT-SCIR, achieved a macro-averaged F1 score of 86.2%. However, performance varied significantly across frameworks:
- Framework Gradient: Accuracy was highest on Flavor 0 (DM at 95.5%) and lowest on Flavor 2 (AMR at 73.4%). This suggests that as representations become more abstract and "further" from the surface text, the mapping task becomes exponentially more difficult for neural models.
- Multi-Task Learning (MTL) Challenges: Although the task encouraged MTL, many teams found that training framework-specific models outperformed joint training. Finding an effective way to share parameters across diverse semantic theories remains an unresolved challenge.
- The ERG Benchmark: The hand-built, rule-based English Resource Grammar (ERG) served as a reference. Notably, the ERG still outperformed data-driven parsers on its native frameworks (DM and EDS), indicating that expert linguistic knowledge remains highly competitive with pure machine learning in specific domains.
Conclusion and Future Directions
The 2019 MRP Shared Task demonstrated the feasibility of unified semantic parsing while highlighting the limitations of current multi-task learning and abstract concept mapping. The primary limitation of the 2019 task was its focus on English-only data. Future iterations aim to expand these efforts to multilingual contexts and incorporate even more diverse frameworks, such as Discourse Representation Structures (DRS), to move closer to a universal semantic encoder.
Summary updated