Introduction
Evaluating the quality of computer-generated text remains a significant challenge in Natural Language Generation (NLG). Traditional metrics often fail to capture the nuances of human language, such as semantic equivalence across different syntactic structures or the impact of grammatical errors. RoMe (Robust Metric) addresses these shortcomings by integrating semantic similarity, syntactic structure, and grammatical correctness into a single, self-supervised neural scorer.
The Limitations of Current Metrics
For years, the field relied on word-overlap metrics like BLEU and ROUGE. These "dumb" metrics struggle with semantic equivalence; for instance, they might fail to recognize that "The dog bit the man" and "The man was bitten by the dog" convey the same meaning. While newer embedding-based metrics like BERTScore and MoverScore utilize contextualized representations, they remain fragile. They often overlook grammatical acceptability and can be easily manipulated by simple text variations, such as word repetition or voice transformations.
The RoMe Framework
RoMe calculates a final quality score by processing three distinct features through a two-layer feed-forward neural network.
1. Enhanced Semantic Similarity
RoMe employs Earth Mover’s Distance (EMD) to measure the cost of transforming a hypothesis sentence into a reference. It introduces Hard Word Alignment to ensure each word in the hypothesis matches at most one word in the reference, effectively penalizing repetition.
Additionally, a soft-penalization constant reduces the penalty for word-order changes common in passive voice, allowing the metric to recognize semantic equivalence across different voices.
2. Semantically Enhanced Tree Edit Distance (TED-SE)
To evaluate syntactic integrity, RoMe compares dependency trees. Unlike standard Tree Edit Distance (TED), TED-SE uses cosine similarity between word embeddings to allow for synonyms, ensuring that lexical variety isn't mistaken for syntactic error.
3. Grammatical Acceptability
A binary classifier, based on BERT-large and trained on the CoLA dataset, predicts the probability of a sentence being grammatically correct.
The final scorer is trained using a self-supervised paradigm on the KELM dataset, where "negative" examples are generated through adversarial attacks to teach the model to distinguish high-quality text from flawed generation.
Key Results and Robustness
RoMe was evaluated across diverse tasks, including data-to-text generation (BAGEL, SFHOTEL, WebNLG) and dialogue systems.
- Higher Correlation: RoMe consistently outperformed BLEU, METEOR, BERTScore, and MoverScore in Spearman correlation with human judgment. In dialogue evaluation, it achieved a correlation of 0.78, significantly higher than BERTScore's 0.62.
- Superior Robustness: In adversarial tests involving passive voice transformations, RoMe maintained a correlation of 0.28, while BLEU plummeted to a nearly useless 0.02.
Conclusion
RoMe represents a significant step toward more human-like evaluation of NLG systems. By combining semantic, syntactic, and grammatical signals, it offers a robust and modular tool for researchers. While currently limited to English due to its reliance on specific dependency parsers, future work aims to expand RoMe to multi-lingual support and refine its performance on domain-specific datasets.
Summary updated