Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning

Emily M. Bender, Jayeol Chun, Dan Flickinger, Andrey Kutuzov, Sebastian Schuster, Milan Straka, Stephan Oepen, Omri Abend, Jan Hajic, Daniel Hershcovich, Marco Kuhlmann, Tim O’Gorman, Nianwen Xue Iii Organizers, Nianwen Xue, Zde Ňka Urešová, Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, W. W. Nelson, Henry Kučera, E. J. Hajic, Jarmila Hajičová, Petr Panevová, Ondřej Sgall, Silvie Bojar, Eva Cinková, Carl Pollard, Ivan A. Sag, Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, Ari Rappoport, Yoav Artzi, Kenton Lee, Luke Zettlemoyer, Kevin Knight, Ulrich Callmeier, Yantao Du, Fan Zhang, Weiwei Sun, Xiaojun Wan, Xun Zhang, Jeffrey Flanigan, Sam Thomson, Jaime Carbonell, Chris Dyer, Noah Smith, Linfeng Song, Dan Gildea, Gabriel Stanovsky, Ido Dagan, Jana Strakov, Elior Sulem, Yuka Tateisi, Akane Yakushiji, Tomoko Ohta, J Tsujii, Zdeka Ureov, Eva Fukov, Jana indlerov, Chuan Wang, Bin Li, Sameer Pradhan, Xiaoman Pan, Ji Heng, Sepp Hochreiter, Jrgen Schmidhuber, Angelina Ivanova, Lilja vrelidOpenalex Unknown 2019 · DOI · OpenAlex

Abstract

This volume provides linguistic, methodological, and technical background to the target representations, mode of operation, and participating systems in a 'system bake-off' for data-driven parsing into graph-structured representations of sentence meaning.

Introduction: Unifying the Tower of Babel

For decades, the field of Natural Language Processing (NLP) has faced a "balkanization" of semantic representations. Different research communities developed distinct frameworks to capture sentence meaning, ranging from logic-heavy English Resource Semantics to concept-focused Abstract Meaning Representation (AMR). Because these frameworks utilized different formalisms (trees vs. graphs), anchoring methods, and evaluation metrics, cross-framework comparison was historically impossible.

The 2019 Shared Task on Cross-Framework Meaning Representation Parsing (MRP) addressed this gap. By establishing a unified pipeline and evaluation metric, the task invited 18 teams to compete in parsing five distinct semantic frameworks, marking a landmark effort in semantic interoperability.

The MRP Framework: A Unified Formal Model

The core innovation of the 2019 task was a unifying formal model that treats all meaning representations as directed graphs. To accommodate the diversity of existing frameworks, the organizers categorized them into three "flavors" based on their relationship to the surface text:

  • Flavor 0 (Bi-lexical Dependencies): Nodes correspond exactly to surface tokens (e.g., DM, PSD).
  • Flavor 1 (Anchored Graphs): Nodes correspond to arbitrary spans, and multiple nodes can overlap or share anchors (e.g., EDS, UCCA).
  • Flavor 2 (Unanchored Graphs): Nodes represent abstract concepts not directly tied to specific words in the sentence (e.g., AMR).

All frameworks were converted into a uniform JSON-based interchange format, allowing for a single, consistent evaluation metric based on "atomic tuples" (nodes, edges, and properties).

Participating systems generally employed one of three architectural strategies:

  1. Transition-Based: Parsers that construct graphs through a sequence of actions (e.g., SHIFT, REDUCE, ADD_EDGE), adapted to handle cycles and reentrancy.
  2. Factorization-Based: Systems that score potential edges between all node pairs and identify the "Maximum Spanning Connected Subgraph."
  3. Composition-Based: Parsers utilizing formal grammars, such as Hyperedge Replacement Grammar, to build meaning compositionally.

A dominant technical trend was the integration of BERT (Bidirectional Encoder Representations from Transformers). Teams transitioning from static embeddings to BERT reported significant F1 score improvements (often 5-10 points), confirming that deep contextual features are vital for resolving semantic ambiguity.

Experimental Setup and Datasets

The task utilized five established "graphbanks": DM, PSD, EDS, UCCA, and AMR. To test generalization, the organizers included out-of-domain data from the Brown Corpus and a newly annotated set of 100 sentences from the novel The Little Prince. This diverse data suite challenged systems to perform beyond their training distributions.

Key Insights and Results

The top-performing system, HIT-SCIR, achieved a macro-averaged F1 score of 86.2%. However, performance varied significantly across frameworks:

  • Framework Gradient: Accuracy was highest on Flavor 0 (DM at 95.5%) and lowest on Flavor 2 (AMR at 73.4%). This suggests that as representations become more abstract and "further" from the surface text, the mapping task becomes exponentially more difficult for neural models.
  • Multi-Task Learning (MTL) Challenges: Although the task encouraged MTL, many teams found that training framework-specific models outperformed joint training. Finding an effective way to share parameters across diverse semantic theories remains an unresolved challenge.
  • The ERG Benchmark: The hand-built, rule-based English Resource Grammar (ERG) served as a reference. Notably, the ERG still outperformed data-driven parsers on its native frameworks (DM and EDS), indicating that expert linguistic knowledge remains highly competitive with pure machine learning in specific domains.

Conclusion and Future Directions

The 2019 MRP Shared Task demonstrated the feasibility of unified semantic parsing while highlighting the limitations of current multi-task learning and abstract concept mapping. The primary limitation of the 2019 task was its focus on English-only data. Future iterations aim to expand these efforts to multilingual contexts and incorporate even more diverse frameworks, such as Discourse Representation Structures (DRS), to move closer to a universal semantic encoder.

Summary updated

Abstract

We are excited (and a little relieved) to present the proceedings from the 2019 Shared Task on Cross-Framework Meaning Representation Parsing (MRP) at the Conference for Computational Language Learning (CoNLL). This volume provides linguistic, methodological, and technical background to the target representations, mode of operation, and participating systems in a 'system bake-off' for datadriven parsing into graph-structured representations of sentence meaning.
BibTeX
@inproceedings{bender2019proceedings,
  author = {Emily M. Bender and Jayeol Chun and Dan Flickinger and Andrey Kutuzov and Sebastian Schuster and Milan Straka and Stephan Oepen and Omri Abend and Jan Hajic and Daniel Hershcovich and Marco Kuhlmann and Tim O’Gorman and Nianwen Xue Iii Organizers and Nianwen Xue and Zde Ňka Urešová and Laura Banarescu and Claire Bonial and Shu Cai and Madalina Georgescu and Kira Griffitt and Ulf Hermjakob and W. W. Nelson and Henry Kučera and E. J. Hajic and Jarmila Hajičová and Petr Panevová and Ondřej Sgall and Silvie Bojar and Eva Cinková and Carl Pollard and Ivan A. Sag and Natalia Silveira and Timothy Dozat and Marie-Catherine de Marneffe and Samuel Bowman and Miriam Connor and John Bauer and Miriam Connor and John Bauer and Omri Abend and Ari Rappoport and Yoav Artzi and Kenton Lee and Luke Zettlemoyer and Shu Cai and Kevin Knight and Ulrich Callmeier and Yantao Du and Fan Zhang and Weiwei Sun and Xiaojun Wan and Yantao Du and Fan Zhang and Xun Zhang and Weiwei Sun and Xiaojun Wan and Jeffrey Flanigan and Sam Thomson and Jaime Carbonell and Chris Dyer and Noah Smith and Linfeng Song and Dan Gildea and Gabriel Stanovsky and Ido Dagan and Milan Straka and Milan Straka and Jana Strakov and Elior Sulem and Omri Abend and Ari Rappoport and Elior Sulem and Omri Abend and Ari Rappoport and Elior Sulem and Omri Abend and Ari Rappoport and Yuka Tateisi and Akane Yakushiji and Tomoko Ohta and J Tsujii and Zdeka Ureov and Eva Fukov and Jana indlerov and Chuan Wang and Bin Li and Nianwen Xue and Chuan Wang and Sameer Pradhan and Xiaoman Pan and Ji Heng and Nianwen Xue and Omri Abend and Ari Rappoport and Sepp Hochreiter and Jrgen Schmidhuber and Angelina Ivanova and Stephan Oepen and Lilja vrelid and Dan Flickinger},
  title = {{Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning}},
  booktitle = {Openalex Unknown},
  year = {2019},
  doi = {10.18653/v1/k19-2},
}
Figure