Self-Consistency of Large Language Models under Ambiguity

Henning Bartsch, Ole Jorgensen, Domenic Rosati, Jason Hoelscher-Obermaier, Jacob Pfau

Abstract

As Large Language Models (LLMs) are integrated into complex decision-making pipelines, ensuring their internal consistency becomes a critical safety and reliability concern. This paper investigates whether LLMs maintain "self-consistency" when faced with ambiguous tasks—specifically, integer sequence completion where multiple mathematical rules could validly apply. By evaluating the OpenAI model suite, researchers found that while consistency improves significantly with model scale (reaching over 80% in GPT-4), models remain poorly calibrated and struggle to accurately judge the consistency of their own responses.

Introduction

LLMs are trained on vast datasets representing a multitude of human perspectives, which often leads them to internalize inconsistent "beliefs" or patterns. A major challenge in AI safety is ensuring that a model remains consistent across different contexts. For example, if a model provides a specific answer to a question, its subsequent explanation should logically lead to that same result.

This study focuses on ambiguity: scenarios where there is no single "correct" answer. Using the task of integer sequence completion (e.g., given 7, 11, 15, is the next number 19 via an arithmetic progression or 15 via a different rule?), the authors explore whether a model's choice of completion in one context matches the underlying rule it provides in another.

Methodology

The researchers evaluated the OpenAI model suite, ranging from text-davinci-003 to gpt-4, using a multi-faceted approach:

  1. Cross-context Consistency: The model was prompted for a sequence completion in one session. In a separate, independent session, it was prompted for the underlying "rule" (expressed as a Python lambda function). The researchers then verified if the model-generated rule actually produced the model-generated completion.
  2. Model-judged Consistency: Models were asked to evaluate their own work by determining if a specific sequence was consistent with a provided rule.
  3. Distributional Analysis: For models where log probabilities were available, the team analyzed the internal probability distribution to see if the model "recognized" alternative valid answers, even if it did not select them during greedy decoding.
  4. Robustness Testing: The study tested consistency across various perturbations, including changing the "speaker" persona (e.g., instructing the model to be "random"), varying number bases (Base-10 vs. Base-2), and adjusting sequence lengths.

The Ambiguous Integer Sequences Benchmark

To facilitate this research, the authors developed the Ambiguous Integer Sequences benchmark. This dataset consists of 140 unambiguous sequences and 57 ambiguous sequences generated from eight function templates (arithmetic, geometric, power, recursive, etc.). A sequence is defined as ambiguous if at least two different functions from the library produce the same initial sequence but different subsequent values.

Key Insights

Consistency as an Emergent Property

The study found that consistency is not explicitly trained for but emerges as models become more capable. GPT-4 demonstrated significantly higher consistency than its predecessors, suggesting that more sophisticated "world models" or internal logic structures naturally lead to higher self-consistency.

The Calibration Paradox

Interestingly, while GPT-4 is the most consistent model, it is also the most poorly calibrated regarding its own performance. It tends to significantly underestimate its own consistency. Conversely, smaller, less capable models often exhibit overconfidence in their consistency.

Internal Multiplicity

Even when a model consistently selects a single answer, its internal probability mass is often distributed across other valid alternatives. This suggests that LLMs compute multiple possible paths simultaneously but use a selection mechanism—likely influenced by Reinforcement Learning from Human Feedback (RLHF)—to converge on a single output for the user.

Resistance to Prompting

Attempts to decrease consistency by instructing the model to "choose randomly" had little effect. The models appear to have a "default" logical path they prefer to follow, regardless of explicit instructions to behave inconsistently.

Results

  • Consistency Scores: Average cross-context consistency ranged from 67% (text-davinci-003) to 82% (GPT-4), far exceeding the random baseline of approximately 8-15%.
  • Accuracy Correlation: There is a strong positive correlation between a model's mathematical accuracy and its ability to remain consistent.
  • Recall Limitations: While models are proficient at identifying if a specific rule is valid (high precision), they struggle to list all possible valid rules for a given ambiguous sequence (low recall).

Conclusion and Future Work

While the study provides a clean mathematical ground truth using integer sequences, the authors acknowledge the need to test these findings in more subjective linguistic domains, such as political or ethical reasoning. Furthermore, the reliance on black-box APIs limited the ability to perform deep distributional analysis on the latest models like GPT-4.

Future research should focus on open-source models to allow for full inspection of internal states. The authors propose that self-consistency checks could eventually serve as a primary method for identifying and invalidating untrustworthy or hallucinated model responses in real-world applications.

Summary updated

BibTeX
@inproceedings{bartsch2023self,
  author = {Henning Bartsch and Ole Jorgensen and Domenic Rosati and Jason Hoelscher-Obermaier and Jacob Pfau},
  title = {{Self-Consistency of Large Language Models under Ambiguity}},
  booktitle = {BlackboxNLP @ EMNLP},
  year = {2023},
  doi = {10.18653/v1/2023.blackboxnlp-1.7},
}
Figure