TL;DR
This paper investigates how Large Language Models (LLMs) represent and process truth values of sentences, particularly in the context of multiple sentences. They discover that while LLMs do have "truth directions" in their latent space, these directions are surprisingly sensitive to irrelevant context, leading to a lack of coherence in truth judgments. The research also provides evidence that these truth-value directions act as causal mediators in the LLM's inference process.
What Problem Does This Paper Address?
Recent research has shown that LLMs possess internal "directions" in their latent representations that correlate with the truthfulness of sentences. These directions can be used to build "probes" that predict sentence truth with high accuracy, leading some to interpret these as models having "knowledge" or "beliefs." However, a crucial question remains: how do these truth-value representations behave when sentences are presented in context, especially when that context might be misleading or irrelevant? The authors argue that for multi-sentence inputs, the primary evaluation metric should be coherence – the logical consistency of truth-value assignments – rather than just accuracy. They aim to understand if LLMs truly grasp truth in context or if their probes are merely picking up on superficial correlations, and whether these truth-value directions are merely descriptive or actively causal in the inference process.
Methods
The core of the methodology involves probing LLMs to extract representations of sentences. They use three established probing methods:
- Mass Mean Probing (MMP): A supervised method that defines the truth-value direction as the difference between the average representations of true and false statements.
- Logistic Regression (LR): A supervised method trained on the difference between negative and positive statement representations.
- Contrast Consistent Search (CCS): An unsupervised method that aims to find a direction separating positive and negative sentence representations. They also introduce Contrast Consistent Reflection (CCR) as a variant of CCS.
To evaluate coherence, they define four error scores (E1-E4) that measure deviations from expected truth-value behavior in different contextual scenarios (entailment, contradiction, irrelevant premises).
A key experimental component is a causal intervention. Here, they directly manipulate the LLM's internal representations of premises by moving them along the identified truth-value directions. They then measure how this intervention affects the probability assigned to a hypothesis.
Datasets
The paper utilizes two primary datasets:
- EntailmentBank: Contains statements with entailment relationships, derived from science questions. This dataset is used to study how LLMs handle factual knowledge and potential prior beliefs.
- SNLI (Stanford Natural Language Inference): A dataset of premise-hypothesis pairs labeled with entailment, contradiction, or neutral relationships, based on image descriptions. This dataset is used to assess how LLMs handle context when the information is not directly grounded in real-world knowledge (as LLMs don't have access to the images).
Key Insights and Ideas
- Context Sensitivity, Not Coherence: LLMs are sensitive to context when judging truth values, but this sensitivity often extends to irrelevant information, leading to a lack of logical coherence. This suggests that simply attributing "belief" to LLMs based on current truth-value probes might be an oversimplification.
- Truth-Value Directions as Causal Mediators: The experiments demonstrate that moving a premise's representation along a truth-value direction causally influences the hypothesis's representation along the same direction. This strongly suggests these directions are not just descriptive but actively mediate the inference process.
- Probing Method Matters: Different probing methods (MMP, LR, CCS) exhibit varying levels of coherence and causal mediation. Logistic Regression, in particular, showed limitations in revealing coherent truth-value judgment.
- Prior Beliefs Aren't Isolated: The "no-prem" probes, intended to capture prior beliefs, still show significant premise sensitivity when evaluated in context. This indicates that LLMs may not represent prior beliefs in a completely orthogonal manner to contextual information.
- Instruction Tuning's Impact: Instruction-tuned models tend to exhibit more errors related to E4 (where the model's own evaluation of context is key), suggesting they are more sensitive to the truthfulness of asserted premises.
Key Results
- High Error Scores: Probes trained on irrelevant contexts (e.g., "no-prem" settings) often result in high error scores (E1, E2), indicating sensitivity to spurious correlations.
- Premise Effect: Hypotheses are significantly more likely to be judged true when entailed by a premise and less likely when contradicted, even when probes are trained without premises.
- Causal Intervention: Moving affirmed premises backward in the truth-value direction led to a decrease in probabilities for entailed hypotheses and an increase for contradicted ones, with changes up to around ten percentage points.
- Model Performance: Larger models (Llama2-13b) did not consistently show better coherence than smaller ones (Llama2-7b).
Limitations and Future Work
- Coherence vs. Accuracy: The paper highlights that high accuracy on truth-value probes doesn't guarantee coherence in contextual reasoning.
- Spurious Correlations: The study acknowledges that probes might pick up on spurious correlations, especially in datasets like SNLI.
- Future Work: The authors suggest investigating whether LLMs occupy a multi-dimensional subspace for truth values, corresponding to different types of beliefs (prior, marginal, conditional). They also propose exploring the representations of meaning relations and the exact mechanisms for incorporating contextual information into truth-value directions. Further work could also focus on developing better methods for evaluating truth-value probes and understanding their failure modes.
Summary updated