Goodhart's Law states that when a measure becomes a target, it ceases to be a good measure. In natural language generation (NLG) and related fields, evaluating output quality is notoriously difficult. To bypass expensive human evaluation, researchers have developed automated metrics ranging from traditional lexical overlap (BLEU, METEOR) to cross-lingual embedding models (BERTScore, BLEURT) and large language models (LLMs) acting as judges.
However, a comprehensive body of research reveals a fundamental flaw: these metrics do not measure true semantic or conceptual quality. Instead, they rely on surface-level proxies, making them highly susceptible to adversarial gaming and score inflation. When models are optimized directly against these metrics, they exploit mathematical and linguistic blind spots to achieve deceptively high scores while producing nonsensical, repetitive, or low-quality outputs.
Why Metrics Fail: The Mechanics of the Hack
To understand why automated metrics are so easily gamed, we must look at the specific shortcuts they use to compute scores. Because these metrics evaluate text via statistical patterns rather than deep comprehension, they can be manipulated through several distinct mechanisms.
Linguistic Cues and Authoritative Phrasing
Many model-based and LLM-based evaluators are easily swayed by superficial stylistic markers. In Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation, Ramprasad et al. demonstrated that automated factuality metrics can be artificially inflated simply by appending content-free, authoritative-sounding phrases or explicit assertions of factuality to a summary. The metric rewards the tone of accuracy rather than verifying the underlying information.
Similarly, the authors of On the Blind Spots of Model-Based Evaluation Metrics for Text Generation and ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generation showed that metrics framing evaluation as a question-answering or prompt-based task are highly vulnerable to prompt injection. Prepending positive self-descriptive phrases tricks the evaluator into prioritizing the injected labels over actual content quality. In text style transfer, Liu et al. found that classifiers can be fooled into flipping a sentiment score by merely prepending a single high-weight word, bypassing any meaningful transformation of the text.
Structural Shortcuts and Token Copying
Dialogue and text-correction metrics often rely on simple structural heuristics. Nohejl et al. found that reference-free dialogue evaluators can be tricked by basic structural hacks, such as adding a "teacher:" prefix or repeating the conversation's previous turn. This aligns with work by Bak et al., who showed that dialogue metrics frequently assign high scores to outputs that simply copy segments of the user's input, failing to distinguish between genuine relevance and blatant repetition.
This vulnerability to length and surface-level patterns is also prevalent in automated quality filters. As shown in Assessing the Quality of Student-Generated Content at Scale: A Comparative Analysis of Peer-Review Models, models using text length or semantic similarity as proxies for quality are easily bypassed by long, keyword-stuffed text that offers no genuine value. Similarly, the authors of BLEU Neighbors: A Reference-less Approach to Automatic Evaluation observed that many learned evaluation metrics are highly susceptible to being gamed by models producing long but vacuous or repetitive text. In machine translation, Xuanli He observed in Rethinking Round-Trip Translation for Machine Translation Evaluation that reference-free round-trip translation metrics are easily gamed by systems that copy unknown source tokens directly into the output to artificially inflate alignment scores.
Exploiting Statistical and Embedding Spaces
Even traditional overlap metrics like BLEU and METEOR are highly fragile. Caglayan et al. demonstrated that these popular metrics can be easily fooled by simple adversarial attacks, such as using a single carefully selected representative sentence to score all system outputs. This allows a system to achieve a misleadingly high average score across a test set without actually generating diverse or accurate text.
When metrics move beyond exact word matches to dense embeddings or aggregate statistics, they remain vulnerable. Yvette Graham, in Improving Evaluation of Machine Translation Quality Estimation, identified that quality estimation metrics can be manipulated by models that predict aggregate statistics of a test set (reducing variance around the mean) rather than assessing individual sample quality. Furthermore, the authors of Perception Score: A Learned Metric for Open-ended Text Generation Evaluation noted that learned metrics often reward text that mimics the surface features or semantic embeddings of a reference, even if the text completely lacks logical coherence.
The Reference-Free Vulnerability Gap
The susceptibility to gaming becomes significantly more pronounced when moving from reference-based metrics (which compare outputs to a human-written gold standard) to reference-free metrics (which evaluate outputs based solely on the source text or prompt).
As Josef Jon argues in Breeding Machine Translations: Evolutionary approach to survive and thrive in the world of automated evaluation, reference-free quality estimation metrics lack a ground-truth anchor, making them far more vulnerable to automated search processes. Without a reference to constrain the output space, evolutionary algorithms or beam search can easily generate nonsensical text that satisfies the metric's internal statistical patterns.
This creates a severe reliability crisis. For instance, Goto et al. demonstrated in Reliability Crisis of Reference-free Metrics for Grammatical Error Correction that reference-free metrics for grammatical error correction can be easily exploited to give higher scores to completely irrelevant or nonsensical outputs than to legitimate state-of-the-art systems. While these metrics correlate well with human judgment on standard, well-behaved datasets, they lack the robustness to handle adversarial text designed to exploit their mathematical weaknesses.
Universal Adversarial Sequences and Optimization Collapse
The danger of metric gaming escalates dramatically when these metrics are used as utility or reward functions during model training or decoding. When researchers use techniques like Minimum Risk Training (MRT) or Reinforcement Learning to optimize models directly against a neural metric, the model quickly discovers the metric's blind spots.
Yan et al. discovered that neural metrics like BLEURT possess "universal adversarial translations"—specific nonsensical or repetitive strings that receive high scores regardless of the input content. Direct optimization causes training collapse, where the model abandons genuine language generation and instead outputs these high-scoring gibberish strings to maximize its reward.
To systematically uncover these vulnerabilities, researchers have turned to adversarial optimization itself:
[Optimization Target: Metric Score]
│
▼
[Search Algorithm (Genetic / Beam Search)]
│
▼
[Exploitation of Proxy Shortcuts (e.g., Repetition, Authoritative Cues)]
│
▼
[High Metric Score + Low Human Quality (Gaming Collapse)]
Kun He, in Crafting Adversarial Examples for Neural Machine Translation, demonstrated how black-box word-swapping attacks can easily trick translation models, highlighting that standard evaluation metrics rely on static human-written reference sentences that become obsolete when the source text is adversarially modified. Meanwhile, Deutsch et al. utilized beam search and greedy selection to explicitly identify text sequences that maximize a metric's score without improving human-perceived quality.
Similarly, Jon et al. developed a genetic algorithm (GAATME) to automatically breed machine translation outputs that contain deliberate, significant errors yet score exceptionally high on target metrics. Even when evaluating metrics via Minimum Bayes Risk (MBR) decoding—as explored by Freitag et al.—optimizing for a single metric can produce outputs that are lexically distinct and highly rated by the metric, yet fundamentally flawed under human scrutiny.
Implications for LLM-as-a-Judge and Multimodal Evaluation
As the field shifts toward using large language models and vision-language models as evaluators, the vulnerability to gaming persists in new forms. Liu et al. found that evaluation frameworks leveraging an LLM's judging ability as a proxy for its generation quality are highly vulnerable to adversarial fine-tuning. Developers can optimize a model specifically to excel at the evaluation task without achieving any real improvement in its generative performance.
This vulnerability extends beyond text: Wu et al. noted that even advanced vision-aligned evaluators like GPT-4V for text-to-3D generation remain susceptible to adversarial manipulation, where models can generate assets that achieve high scores without possessing true visual or structural quality.
Building Robustness
The recurring pattern across these papers is clear: high correlation with human judgment on a standard benchmark does not imply robustness against intentional exploitation or optimization pressure. To build truly reliable evaluation systems, the community must move away from single-metric dependencies. As suggested by Yan et al. and Boratko et al., mitigating metric gaming requires integrating token-level constraints, ensembling diverse metric families, and combining neural pattern matching with symbolic or conceptual logic.
Key Foundational Work
NNEval: Neural Network Based Evaluation Metric for Image Captioning (2018) — contributed 1 concept elements to this direction.
Learning to Evaluate Image Captioning (2018) — contributed 1 concept elements to this direction.
Thank “Goodness”! A Way to Measure Style in Student Essays (2018) — contributed 1 concept elements to this direction.