TL;DR
This paper addresses the challenge of evaluating cross-lingual Abstract Meaning Representation (AMR) parsers, which currently rely on English gold standards despite parsing non-English text. The authors propose and evaluate multilingual adaptations of existing AMR evaluation metrics, finding that a new metric, XS2match, best correlates with human judgments of cross-lingual similarity. This new metric offers a more accurate way to assess the performance of parsers that aim to capture meaning faithfully across languages.
What Problem Does This Paper Address?
Abstract Meaning Representation (AMR) is a graph-based formalism used to represent the meaning of sentences. Cross-lingual AMR parsers aim to generate these AMR graphs from sentences in languages other than English. However, a significant problem in evaluating these parsers is the lack of multilingual AMR evaluation metrics. Consequently, existing cross-lingual parsers are typically evaluated by comparing their output (which is often in English-focused AMR) against gold-standard English AMR annotations, even when the input sentence is in a different language. This practice is problematic because the structure of AMR graphs is known to be influenced by the source language's syntax and semantics. This means that simply comparing to English AMR might not accurately reflect how well the parser captures the meaning of the original non-English sentence.
Methods
The authors propose three multilingual adaptations of existing monolingual AMR evaluation metrics:
- XSmatch: This metric adapts the standard Smatch metric by translating individual tokens within the non-English AMR graphs into English before comparison. They use the EasyNMT package for this translation.
- XSemBleu: This metric adapts the SemBleu metric (based on the machine translation BLEU score) by translating tokens in one of the AMRs. It also involves truncating tokens after translation to handle potential discrepancies.
- XS2match: This metric adapts the S2match metric by leveraging LaBSE (Language-agnostic BERT Sentence Embedding) embeddings. Instead of relying on pre-defined text files of embeddings, it elicits constant tensors for words, preprocesses them, and encodes them using LaBSE. This approach avoids potential biases from neural machine translation tools used by parsers.
They then evaluate these proposed metrics by comparing their correlation with human judgments of cross-lingual sentence similarity.
Datasets
- AMR 2.0 dataset: Used for English AMR annotations.
- AMR 2.0 - Four Translations dataset: Provides parallel sentences and their corresponding English AMR annotations for Mandarin Chinese, Spanish, Italian, and German.
- Spanish AMR corpus (from Wein et al., 2022): Used to create Spanish gold AMR annotations for evaluation.
- Human Judgments: The authors collected human judgments on 100 Spanish-English and 150 Mandarin Chinese-English sentence pairs. These judgments were collected to serve as a ground truth for evaluating the proposed AMR metrics.
Key Insights and Ideas
- Language Matters: The core insight is that source language significantly impacts AMR structure, making monolingual English evaluation metrics insufficient for cross-lingual AMR parsing.
- XS2match is King: The proposed XS2match metric, which uses LaBSE embeddings, shows the highest correlation with human judgments of cross-lingual sentence similarity, especially for Chinese-English pairs.
- Translation is Beneficial (but with caveats): Translating AMR components (as in XSmatch and XSemBleu) can improve performance, but XS2match's approach of using cross-lingual embeddings is more robust and less prone to biases from specific translation tools.
- Monolingual Smatch as a Proxy: While not ideal, monolingual Smatch against English gold AMRs can still serve as a reasonable proxy for ranking parser performance in the absence of in-language gold AMRs.
- Chinese-English Challenges: Evaluating cross-lingual AMR for Chinese-English pairs is generally harder, likely due to greater structural dissimilarity between the languages and potential limitations in machine translation and embedding models for Chinese.
Key Results
- XS2match Correlation: XS2match achieved the highest Pearson's correlation with human judgments of sentence similarity (0.51 for Chinese-English and 0.80 for Spanish-English).
- Metric Performance: XSmatch also performed well, while XSemBleu showed moderate correlation. BERTscore, used as a reference, also showed strong correlations, validating the human judgments.
- Parser Evaluation: When evaluating state-of-the-art cross-lingual AMR parsers, XS2match (with Spanish gold AMRs) provided a more accurate and fairer comparison than monolingual Smatch (with English gold AMRs).
- Translate then Parse (TP) Dominance: The "Translate then Parse" approach, which translates the source sentence to English before parsing, consistently achieved the highest similarity scores across metrics, suggesting the effectiveness of high-quality machine translation as a preprocessing step.
Limitations and Future Work
- Limited Gold Data: The authors acknowledge that the evaluation of cross-lingual AMR parsers will become more robust with the future production of more non-English gold AMR annotations.
- Bias in SemBleu: XSemBleu's performance was hampered by an undesirable bias in the metric that could not be overcome without introducing another bias.
- Future Work: The paper suggests exploring additional multilingual evaluation metrics and continuing to develop robust methods for cross-lingual AMR evaluation as more in-language gold data becomes available. They also highlight the potential of using machine translation as a preprocessing step for cross-lingual parsing.
Summary updated