TL;DR
This paper introduces RCScore, a novel metric that evaluates Large Language Models (LLMs) not just on accuracy but also on their consistency across different ways of asking the same question. They found that LLMs can be surprisingly sensitive to instruction phrasing, and this inconsistency often correlates with lower accuracy, suggesting RCScore can be a valuable proxy for model reliability.
What Problem Does This Paper Address?
Current evaluations of LLMs often rely on a single, fixed way of asking questions (instruction templates) and measure performance primarily through accuracy. This approach overlooks a critical aspect: how sensitive LLMs are to the way a question is phrased. Real-world applications require models that can understand and respond reliably even when instructions vary slightly. This paper argues that traditional benchmarks fail to capture this nuance, potentially leading to an incomplete understanding of LLM capabilities and reliability. They aim to provide a more robust evaluation by quantifying how consistent an LLM's responses are when the same underlying problem is presented with different stylistic variations in the instructions.
Methods
The core of this paper is the introduction of RCScore, a multi-dimensional framework designed to measure response consistency across different instruction styles. RCScore breaks down consistency into three key dimensions:
- Structurality: Measures how consistently the syntactic patterns (like part-of-speech tags and dependency relations) are preserved across responses.
- Lexicality: Assesses the consistency in vocabulary and terminology used in the responses.
- Coherence: Evaluates the logical flow and organizational similarity of the responses.
To apply RCScore, the authors developed a method called Cross-Response Similarity (CRS). This involves taking a single problem and generating responses using multiple, systematically varied instruction styles (e.g., declarative, interrogative, exclamative, imperative). CRS then quantifies the similarity between these responses along the RCScore dimensions. This allows them to measure a model's "stylistic self-consistency."
Datasets
The paper uses four established reasoning benchmarks:
- AIME: American Invitational Mathematics Examination (competitive math problems).
- MATH-500: A subset of problems from the MATH benchmark.
- GPQA-Diamond: A subset of the GPQA benchmark, adapted for pure reasoning by removing answer options.
- GSM8K: Grade school math word problems requiring multi-step reasoning.
Key Insights and Ideas
- Instruction Style Matters (A Lot): LLMs are surprisingly sensitive to how instructions are phrased. Even minor stylistic changes can lead to significant variations in accuracy, sometimes by over 16%.
- Consistency is Key: Models that produce stylistically similar responses across different instruction styles (high CRS) tend to achieve higher task accuracy. This suggests that consistency is a strong indicator of a model's underlying reliability and reasoning ability.
- Greedy Decoding is More Consistent: Deterministic decoding strategies (like greedy search) generally lead to more consistent outputs across instruction styles compared to more exploratory methods (like beam search with temperature).
- Larger Models are More Consistent: Generally, larger LLMs exhibit higher stylistic consistency, suggesting that scale contributes to robustness against prompt variations.
- Task Complexity Amplifies Sensitivity: More complex reasoning tasks (like AIME) show higher style sensitivity than simpler ones (like GSM8K).
Key Results
- Instruction style variations caused accuracy shifts of up to 16.7 percentage points across models and benchmarks.
- The aggregated RCScore (CRS) showed a significant positive correlation with mean task accuracy (Pearson's r > 0.65 for beam search, r ≈ 0.79 for greedy search).
- Lexicality was particularly strongly correlated with accuracy.
- Greedy search consistently yielded higher CRS values (indicating greater consistency) than beam search, often by 7-13%.
- Larger models (e.g., LLaMA 3.3-70B, Qwen 2.5-72B) generally showed higher CRS values.
Limitations and Future Work
Limitations:
- The current framework relies on a predefined set of four linguistic styles and specific English NLP tools (dependency parsing, TF-IDF). Its applicability to other languages or broader stylistic variations (like code-switching) is unexplored.
- The computational cost, while minimized, could still be a factor for extremely large-scale evaluations.
- The focus on mathematical and reasoning tasks may not generalize to more subjective domains like creative writing or emotional support.
- The weights for aggregating RCScore dimensions are currently uniform and could be task-dependent.
- Validation through human evaluation is lacking.
Future Work:
- Leverage CRS as a more direct indicator of robust reasoning capabilities, hypothesizing that high CRS implies genuine reasoning skills rather than superficial pattern matching.
- Use RCScore's dimensional feedback to refine model architectures and training objectives for improved response consistency.
- Inform prompt engineering practices by identifying styles that elicit consistently well-formed and accurate responses.
- Extend the framework to diverse generative tasks, modalities, and cross-lingual analyses.
- Investigate task-dependent or empirically derived weighting schemes for RCScore dimensions.
Summary updated