Truth is Universal: Robust Detection of Lies in LLMs

Lennart Bürger, Fred A. Hamprecht, Boaz Nadler

Introduction

As Large Language Models (LLMs) gain autonomy, the risk of strategic deception—lying to achieve specific goals—becomes a critical safety concern. While previous attempts to build "lie detectors" focused on internal activations, they often failed to generalize. For instance, a detector trained on affirmative statements like "The capital of France is Paris" would frequently fail on negated counterparts like "The capital of France is not Berlin." This research identifies a robust, two-dimensional subspace within LLM internal representations that disentangles "truth" from "grammatical polarity," leading to a highly accurate and universal lie detection method.

The 2D Geometry of Truth

The core discovery of this work is that truth in LLMs is not represented by a single vector but lives within a 2D subspace. Previous failures in linear probing occurred because simple classifiers learned a "tangled" direction that conflated factual truth with sentence structure (affirmative vs. negated).

Visualization of the 2D truth subspace, showing the general truth axis ($t_G$) and the polarity axis ($t_P$).

The researchers identified two primary axes:

  1. General Truth ($t_G$): An axis that consistently points from false to true across all statement types.
  2. Polarity ($t_P$): An axis that points from false to true for affirmative statements but flips (true to false) for negated ones.

By explicitly separating these directions using the TTPD (Training of Truth and Polarity Direction) pipeline, the authors can project activations onto this 2D plane and use a simple logistic regression classifier to detect deception with high reliability.

Experimental Setup and Datasets

The study utilized a comprehensive dataset of approximately 45,000 labeled true/false statements. These spanned various categories, including:

  • Factual Topics: Cities, translations, chemical elements, and animal classifications.
  • Logical Variations: Negated versions of facts, as well as complex conjunctions ("and") and disjunctions ("or").
  • Cross-Lingual: English statements translated into German to test representation universality.
  • Deceptive Scenarios: Role-playing prompts where models were incentivized to lie (e.g., an AI agent hiding property defects).

Results and Universality

The TTPD method achieved an average accuracy of 94% in distinguishing truth from falsehood. Crucially, the 2D structure was found to be consistent across diverse model families, including Gemma, LLaMA2, Mistral, and LLaMA3.

AUROC scores across datasets, demonstrating the stability of the general truth direction ($t_g$) compared to the polarity-dependent direction ($t_p$).

As shown in the performance metrics, the $t_G$ direction remains highly predictive (AUROC near 1.0) across both affirmative and negated datasets. In contrast, the $t_P$ direction's performance drops to near zero for negated sets, confirming its dependence on grammatical polarity. In complex role-playing scenarios, TTPD maintained a 93.8% accuracy, significantly outperforming baseline methods like Contrast Consistent Search (CCS).

Discussion and Future Work

The findings suggest that as LLMs scale, they converge on a "Platonic" internal representation of reality that persists even when the model is prompted to output a lie. However, several avenues for future research remain:

  • Non-linearity: While linear subspaces are effective, non-linear structures might offer even higher precision.
  • Abstract Truths: The current work focuses on factual statements; extending this to moral or subjective truths is a key challenge.
  • Intervention: Future work could explore using these identified axes to actively intervene in model processing, potentially "forcing" honesty by shifting activations along the $t_G$ axis.

Summary updated

BibTeX
@inproceedings{burger2024truth,
  author = {Lennart Bürger and Fred A. Hamprecht and Boaz Nadler},
  title = {{Truth is Universal: Robust Detection of Lies in LLMs}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=1Fc2Xa2cDK},
}
Figure