TL;DR
This paper presents a novel human evaluation of AMR-to-English generation systems, moving beyond solely relying on automated metrics like BLEU. Their findings reveal that while automated metrics generally rank systems correctly, human evaluation offers crucial insights into nuanced strengths and weaknesses, highlighting common error types like hallucination and omission.
What Problem Does This Paper Address?
Abstract Meaning Representation (AMR) is a powerful way to represent sentence meaning as a graph, but generating natural language (English) from these graphs is challenging. Evaluating these generation systems is particularly tricky because there can be many valid ways to express the same meaning. Current practice heavily relies on automated metrics like BLEU, which compare generated text to a single human-written reference. However, research suggests these metrics are often poor proxies for human judgment in natural language generation (NLG), and specifically for AMR generation. This paper addresses this gap by conducting a comprehensive human evaluation to provide a more reliable assessment of AMR generation systems and to identify common error patterns.
Methods
The authors designed a new human evaluation methodology that moves away from direct comparison to a single reference. Instead, they employed a direct assessment approach where annotators rated sentences on two independent scales: fluency (how natural the English sounds) and adequacy (how well the sentence captures the meaning of the AMR graph). This referenceless design was chosen to avoid biasing annotators towards a specific phrasing. Additionally, for adequacy, annotators could select from predefined error types: "cannot understand meaning," "information from the AMR is missing," and "information not present in the AMR is added." The evaluation involved nine annotators, all with prior AMR experience, who rated a subset of sentences from a standard AMR dataset.
Datasets
The evaluation used the test set from the LDC2017T10 AMR dataset. To ensure comparability across systems, they focused on AMRs present in both the 2015 and 2017 versions of the dataset. Sentences with multisentence root relations were excluded due to their complexity and length. Ultimately, 998 out of 1371 original sentences were deemed eligible, and a random sample of 100 of these were used for the main human evaluation.
Key Insights and Ideas
- Human evaluation is indispensable: While automated metrics like BLEU, METEOR, and BERTScore generally align with human rankings, they miss crucial nuances. Human judgment is essential for understanding why a system performs well or poorly.
- Fluency vs. Adequacy are distinct but related: Annotators could largely separate judgments of fluency and adequacy, though a moderate correlation (0.58 in the main survey) indicates they are not entirely independent.
- Neural systems are prone to hallucination and omission: A significant finding is that neural AMR generation systems frequently omit information from the AMR or "hallucinate" (add) information not present. This is a major barrier to practical application.
- Rule-based systems have different error profiles: Manning's rule-based system, while less prone to omission and hallucination, produced significantly more incomprehensible sentences, often due to a lack of function words.
- Anonymization and repetition are fluency pitfalls: For neural systems, anonymizing infrequent concepts (using
<unk>) and unhumanlike repetition of words or phrases were common sources of low fluency scores.
Key Results
The study evaluated five AMR generation systems: Konstas, Zhu, Ribeiro, Guo, and Manning.
- System Ranking: Konstas consistently achieved the highest average fluency and adequacy scores, followed by Zhu. Manning performed the worst. Ribeiro and Guo were in the middle, with Ribeiro slightly better on fluency and Guo on adequacy.
- Error Analysis:
- Low Adequacy: 33% of low-adequacy sentences were incomprehensible, 37% had missing information, and 18% had added information.
- Low Fluency: 43% of low-fluency sentences were also incomprehensible. Manning was responsible for over half of all low-fluency sentences.
- Metric Correlation: BERTScore showed the strongest correlation with human judgments for both adequacy and fluency, outperforming BLEU and METEOR.
- Inter-Annotator Agreement (IAA): IAA for numeric fluency and adequacy scores was moderate to high (Spearman's Rho of 0.78 and 0.67, respectively). However, IAA for adequacy error types was considerably lower (Cohen's Kappa ranging from 0.28 to 0.53), suggesting a need for clearer annotation guidelines for these specific error categories.
Limitations and Future Work
The authors acknowledge that the inter-annotator agreement for error types was not as high as desired, indicating that the guidelines for identifying these errors could be clearer. They suggest that future studies should refine these guidelines. They also note that while automatic metrics generally ranked systems correctly, human evaluation provided a more nuanced understanding of system strengths and weaknesses. The paper strongly recommends that researchers in AMR generation and other NLG tasks continue to supplement automatic metrics with human evaluation whenever possible to gain a more complete picture of system performance. Future work should focus on addressing the identified error types, particularly hallucination and unnecessary repetition.
Summary updated