RoMe: A Robust Metric for Evaluating Natural Language Generation

Md. Rashad Al Hasan Rony, Liubov Kovriguina, Debanjan Chaudhuri, Ricardo Usbeck, Jens Lehmann

Introduction

Evaluating the quality of computer-generated text remains a significant challenge in Natural Language Generation (NLG). Traditional metrics often fail to capture the nuances of human language, such as semantic equivalence across different syntactic structures or the impact of grammatical errors. RoMe (Robust Metric) addresses these shortcomings by integrating semantic similarity, syntactic structure, and grammatical correctness into a single, self-supervised neural scorer.

The Limitations of Current Metrics

For years, the field relied on word-overlap metrics like BLEU and ROUGE. These "dumb" metrics struggle with semantic equivalence; for instance, they might fail to recognize that "The dog bit the man" and "The man was bitten by the dog" convey the same meaning. While newer embedding-based metrics like BERTScore and MoverScore utilize contextualized representations, they remain fragile. They often overlook grammatical acceptability and can be easily manipulated by simple text variations, such as word repetition or voice transformations.

The RoMe Framework

RoMe calculates a final quality score by processing three distinct features through a two-layer feed-forward neural network.

1. Enhanced Semantic Similarity

RoMe employs Earth Mover’s Distance (EMD) to measure the cost of transforming a hypothesis sentence into a reference. It introduces Hard Word Alignment to ensure each word in the hypothesis matches at most one word in the reference, effectively penalizing repetition.

Hard Word Alignment prevents multiple matches to penalize word repetition.

Additionally, a soft-penalization constant reduces the penalty for word-order changes common in passive voice, allowing the metric to recognize semantic equivalence across different voices.

2. Semantically Enhanced Tree Edit Distance (TED-SE)

To evaluate syntactic integrity, RoMe compares dependency trees. Unlike standard Tree Edit Distance (TED), TED-SE uses cosine similarity between word embeddings to allow for synonyms, ensuring that lexical variety isn't mistaken for syntactic error.

3. Grammatical Acceptability

A binary classifier, based on BERT-large and trained on the CoLA dataset, predicts the probability of a sentence being grammatically correct.

The final scorer is trained using a self-supervised paradigm on the KELM dataset, where "negative" examples are generated through adversarial attacks to teach the model to distinguish high-quality text from flawed generation.

Key Results and Robustness

RoMe was evaluated across diverse tasks, including data-to-text generation (BAGEL, SFHOTEL, WebNLG) and dialogue systems.

Correlation matrix showing RoMe's strong performance against human judgments (Inf, Nat, Qual) compared to baseline metrics.
  • Higher Correlation: RoMe consistently outperformed BLEU, METEOR, BERTScore, and MoverScore in Spearman correlation with human judgment. In dialogue evaluation, it achieved a correlation of 0.78, significantly higher than BERTScore's 0.62.
  • Superior Robustness: In adversarial tests involving passive voice transformations, RoMe maintained a correlation of 0.28, while BLEU plummeted to a nearly useless 0.02.

Conclusion

RoMe represents a significant step toward more human-like evaluation of NLG systems. By combining semantic, syntactic, and grammatical signals, it offers a robust and modular tool for researchers. While currently limited to English due to its reliance on specific dependency parsers, future work aims to expand RoMe to multi-lingual support and refine its performance on domain-specific datasets.

Summary updated

Abstract

Evaluating Natural Language Generation (NLG) systems is a challenging task. Firstly, the metric should ensure that the generated hypothesis reflects the reference's semantics. Secondly, it should consider the grammatical quality of the generated sentence. Thirdly, it should be robust enough to handle various surface forms of the generated sentence. Thus, an effective evaluation metric has to be multifaceted. In this paper, we propose an automatic evaluation metric incorporating several core aspects of natural language understanding (language competence, syntactic and semantic variation). Our proposed metric, RoMe, is trained on language features such as semantic similarity combined with tree edit distance and grammatical acceptability, using a self-supervised neural network to assess the overall quality of the generated sentence. Moreover, we perform an extensive robustness analysis of the state-of-the-art methods and RoMe. Empirical results suggest that RoMe has a stronger correlation to human judgment over state-of-the-art metrics in evaluating system-generated sentences across several NLG tasks.
BibTeX
@inproceedings{rony2022rome,
  author = {Md. Rashad Al Hasan Rony and Liubov Kovriguina and Debanjan Chaudhuri and Ricardo Usbeck and Jens Lehmann},
  title = {{RoMe: A Robust Metric for Evaluating Natural Language Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.387},
}
Figure