Abstract
Large Language Models (LLMs) often exhibit brittleness when faced with minor, non-malicious input variations such as typos or rephrasing. This paper introduces a normalized version of Cohen’s $h$ ($\tilde{H}$) as a more reliable metric for measuring LLM robustness compared to the standard Performance Drop Rate (PDR). By utilizing an arcsine transformation, $\tilde{H}$ provides a symmetric, stable, and interpretable measure that accounts for both performance gains and losses across all score ranges. Evaluation across multiple models and datasets demonstrates that while frontier models like Llama 3 and Mixtral achieve high accuracy, they remain sensitive to linguistic structure, particularly paraphrasing.
The Challenge of LLM Brittleness
While adversarial attacks on LLMs are a significant research focus, real-world failures often stem from "natural" perturbations—simple typos, punctuation changes, or alternative phrasings. The industry standard for quantifying this sensitivity, the Performance Drop Rate (PDR), suffers from two critical mathematical flaws:
- Asymmetry: Performance gains and losses are weighted inconsistently, skewing aggregate data. For instance, a performance increase from 0.1 to 0.8 results in a -700% PDR, while a decrease from 0.8 to 0.1 results in only 87.5%.
- Undefined Values: PDR is undefined when the baseline score is zero, leading to biased results as these cases are often discarded by researchers.
To address these limitations, we propose using a normalized version of Cohen’s $h$ ($\tilde{H}$), a statistical effect size metric. By applying an arcsine transformation to performance scores: $$H = 2 \cdot (\arcsin\sqrt{score_p} - \arcsin\sqrt{score_o})$$ where $score_o$ and $score_p$ represent original and perturbed performance respectively. The authors normalize this value by dividing by $\pi$ to bound the metric ($\tilde{H}$) between -1 and 1 for easier comparison.
Experimental Framework
We evaluated several open-weight models, including Llama 2/3, Mixtral, and IBM’s Granite, across three datasets: PopQA (factual questions), SIGA (social identity group abuse classification), and BoolQ (reading comprehension). These datasets were expanded with three categories of perturbations:
- Superficial (S): Non-semantic changes like casing, typos, or extra whitespace.
- Paraphrase (P): Semantically equivalent rephrasings.
- Distraction (D): Irrelevant text snippets added to the context.
Results and Analysis
Our analysis reveals that accuracy does not necessarily equate to robustness. High-performing models like Llama3-70B and Mixtral-8x7B still show significant sensitivity to perturbations, particularly paraphrasing, which caused the sharpest performance drops across all datasets. This suggests that LLMs remain overly sensitive to linguistic structure rather than just underlying meaning.
Key Findings
- Metric Stability: $\tilde{H}$ correlates strongly with PDR ($r \approx 0.995$) when the original score is 1.0, but handles edge cases (original scores of 0) much more effectively.
- Volatility Detection: The absolute value of the metric ($A\tilde{H}$) reveals hidden instability where models change their predictions in both directions—getting some right that they previously got wrong, and vice versa—a phenomenon PDR fails to capture.
- Model Comparisons: Flan-T5-XXL emerged as highly robust despite lower raw accuracy. Conversely, smaller models like Granite-13B demonstrated high robustness relative to their baseline performance.
Conclusion
The proposed $\tilde{H}$ metric offers a mathematically sound alternative for benchmarking LLM robustness in everyday scenarios. While frontier models lead in accuracy, their sensitivity to linguistic structure remains a hurdle for reliable real-world deployment. Future research will focus on extending this metric to continuous scores (e.g., BLEU, ROUGE) and exploring more complex paraphrasing techniques to further probe model brittleness.
Summary updated