HAUSER: Towards Holistic and Automatic Evaluation of Simile Generation

Qianyu He, Yikai Zhang, Jiaqing Liang, Yuncheng Huang, Yanghua Xiao, Yunwen Chen

Introduction

Simile generation (SG) is a vital task in creative AI, transforming literal statements into evocative figurative language. For example, "He yelps" might become "He yelps like a wounded buffalo." However, evaluating these generations automatically is a significant hurdle. Traditional metrics like BLEU and ROUGE, which rely on n-gram overlap with a reference sentence, are poorly suited for the open-ended nature of figurative writing. A high-quality simile might use a completely different "vehicle" (the object of comparison) than the reference, leading to a low score despite its excellence. Furthermore, these metrics treat all words equally, failing to prioritize the semantic weight of the topic and vehicle.

Comparison of traditional metrics vs. human judgment in simile evaluation.

To address these limitations, we introduce HAUSER (Holistic and AUtomatic evaluation system for Simile gEneRation). HAUSER moves beyond simple string matching to evaluate similes across five dimensions grouped into three perspectives: Quality, Creativity, and Informativeness.

The HAUSER Framework

HAUSER evaluates similes through a multi-faceted lens, leveraging large-scale knowledge bases and pre-trained models.

1. Quality

The quality of a simile is assessed through three criteria:

  • Relevance: Using MAPS-KB, a million-scale simile knowledge base, HAUSER calculates how often a topic and vehicle co-occur in literature, ensuring the comparison is grounded in shared properties.
  • Logical Consistency: A pre-trained MNLI model checks if the simile contradicts the original literal meaning.
  • Sentiment Consistency: An SST-2 classifier ensures the figurative sentence maintains the emotional valence of the original literal input.

2. Creativity

To avoid rewarding clichés like "white as snow," HAUSER measures creativity by analyzing the frequency of the vehicle in a large corpus. Rarer, "long-tail" vehicles that remain relevant receive higher scores, encouraging the generation of novel comparisons.

3. Informativeness

This metric rewards descriptive richness. HAUSER calculates the average length of extracted vehicles, favoring detailed comparisons (e.g., "like the eyes of an angry cat") over generic ones.

Experimental Results

The effectiveness of HAUSER was validated against a human-rated benchmark of 150 simile candidates. The results demonstrate that HAUSER significantly outperforms traditional metrics in correlating with human judgment:

  • Quality: Achieved a Pearson correlation of 0.320, a 6.5% improvement over BLEU2 and nearly triple that of BERTScore.
  • Creativity: Reached a correlation of 0.592, a 27.3% jump over standard diversity metrics.
  • Informativeness: Showed a very strong correlation of 0.798, confirming vehicle length as a robust proxy for content richness.

In recommendation tasks, HAUSER's rankings of "Top 1" candidates aligned much more closely with human experts than any previous automated method.

Conclusion

HAUSER represents a significant step toward automated evaluation that respects the nuances of creative writing. By focusing on semantic relationships rather than literal string overlap, it provides a more accurate and holistic measure of simile quality. Future work will explore reference-free metrics to further reduce dependency on external knowledge bases and improve computational efficiency.

Summary updated

BibTeX
@inproceedings{he2023hauser,
  author = {Qianyu He and Yikai Zhang and Jiaqing Liang and Yuncheng Huang and Yanghua Xiao and Yunwen Chen},
  title = {{HAUSER: Towards Holistic and Automatic Evaluation of Simile Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.702},
}
Figure