TL;DR
This paper introduces a novel framework to systematically evaluate the reliability of value reasoning in small Large Language Models (sLLMs). The findings reveal significant inconsistencies in how sLLMs handle value-based queries, highlighting a critical gap in their ability to provide stable and trustworthy responses, even when their self-reported confidence is high.
What Problem Does This Paper Address?
As Large Language Models (LLMs) become increasingly integrated into various applications, ensuring their alignment with human values is paramount. However, simply instilling values isn't enough; the models must also demonstrate stable and consistent reasoning based on these values across diverse contexts. This paper addresses the critical gap of understanding the reliability of value reasoning in small LLMs (sLLMs), which are becoming prevalent due to their lower computational demands. The authors argue that existing research has largely focused on large models or general robustness, neglecting the specific challenges of value reasoning stability in smaller models, which might be more susceptible to information loss during compression techniques. Inconsistent value-based responses can erode user trust and lead to unpredictable behavior, posing potential societal risks.
Methods
The authors propose a multidimensional evaluation framework called Value-Reasoning Reliability (VRR). This framework comprises three core tasks:
- Repetition Consistency: Assesses how consistently a model responds to the same prompt multiple times.
- Interaction Stability: Evaluates the model's ability to maintain its stance when subjected to persuasive arguments from another agent (simulated by the same model).
- Open-Ended Expression Consistency: Checks if the underlying justifications for a model's value choices remain stable across multiple free-form generations.
To enhance the rigor of these tasks, the framework incorporates a Data Augmentation module. This module generates perturbed versions of standard prompts by introducing paraphrasing, spelling errors, and modifications to answer options. This simulates real-world input imperfections and tests model robustness. Additionally, the framework measures self-reported confidence scores to explore the relationship between a model's perceived certainty and its actual reasoning stability.
Datasets
The study utilizes two established datasets as foundational elements:
- PVQ40 (Portrait Values Questionnaire-40): Derived from Schwartz's theory of ten basic human values, this dataset assesses how similar individuals are to described portraits. It has been adapted for evaluating value expression in LLMs.
- INVP (Investigating Value Priorities): This dataset simulates decision-making processes in social scenarios, also grounded in Schwartz's value theory. It presents binary decisions involving trade-offs between conflicting values.
The authors also introduce newly constructed perturbed datasets by applying data augmentation techniques (paraphrasing, spelling errors, and option modifications) to the original PVQ40 and INVP datasets. These augmented datasets are designed to simulate real-world user input variations and test model robustness.
Key Insights and Ideas
- sLLMs struggle with value reasoning stability: Despite advancements, sLLMs exhibit significant variability in their value-based responses, even under identical prompts. This suggests a lack of a robust internal mechanism for consistent value reasoning.
- Self-reported confidence is a poor indicator of stability: Models often report high confidence even when their responses are inconsistent or random. This disconnect cautions against over-relying on a model's self-assessment of its value judgments.
- Prompt variations significantly impact sLLM behavior: Even minor changes like paraphrasing or spelling errors can lead to noticeable shifts in value judgments, particularly in the self-awareness tasks.
- Open-ended justifications are less stable than choices: While models might make consistent choices, the underlying reasoning provided in free-form text can be highly variable, indicating a superficial understanding or a lack of stable internal "reasoning processes."
- Mistral-7B shows promising consistency, but not immunity: While Mistral-7B generally performs better in terms of consistency, it still shows a substantial drop in stability after prompt perturbations, highlighting that no current sLLM is fully robust.
Key Results
The study evaluated four mainstream sLLMs: GLM4-9B, Llama-3-8B, Mistral-7B, and Qwen2.5-7B.
- Repetition Consistency: Mistral-7B demonstrated the highest consistency on the standard dataset, maintaining perfectly consistent choices across ten trials. In contrast, GLM4-9B showed extremely low consistency with a 100% flip rate. Llama-3-8B and Qwen2.5-7B also displayed low consistency.
- Robustness to Perturbations: Even Mistral-7B showed a substantial drop in consistency (PRR of 20.51%) after minor prompt perturbations in self-awareness tasks. Llama-3-8B exhibited pronounced instability when answer content or option order was altered in value-based decision-making tasks.
- Persuasion Task: Models generally fell into two extremes: either maintaining a completely stable choice or showing significant change after just one round of persuasion. Mistral-7B achieved the highest proportion of completed ten-round interactions, indicating stronger decision stability.
- Open-Ended Expression Consistency: Models exhibited low stance consistency across ten responses in open-ended tasks, with Mistral-7B showing only 10% consistency in self-awareness tasks.
- Confidence vs. Stability: The correlation between model confidence scores and value stability was generally low, suggesting that high confidence does not reliably predict stable value judgments.
Limitations and Future Work
The authors acknowledge several limitations:
- Scope of sLLMs: The study evaluated only four sLLMs, and findings may not generalize to all sLLMs or larger models.
- Choice of Base Datasets: The reliance on Schwartz's value theory represents a specific operationalization of values, and other value systems or cultural contexts might yield different outcomes.
- Computational Resources: The limitations in computational power affected the number of models and variants that could be evaluated.
Future work could explore:
- Evaluating a broader range of sLLMs and larger models.
- Investigating value systems beyond Schwartz's theory.
- Developing more sophisticated methods to improve the stability and reliability of value reasoning in LLMs.
- Further research into the "value-action gap" and how to ensure models' actual behavior aligns with their stated values.
Summary updated