Graded Veracity from a Binary Truth Direction, Read Clause by Clause

Language models carry an internal signal for "this statement is true." Probes trained to find it transfer well across question-answering datasets, and a single "falsehood direction" learned on political claims carries over to other fact-checking corpora. Real misinformation is rarely just true or false, though. Fact-checkers rate claims as "half true" or "mostly false," and those intermediate ratings are exactly where automated systems fail. Every existing internal-signal detector is trained on binary labels and scores a claim as one pooled vector, so we do not know whether the model's internal geometry holds anything about degrees of truth.

This note proposes a simple, testable account of where graded veracity lives. A compound claim earns an intermediate rating because some of its parts are false and others true. The prediction is that reading a binary truth direction at each clause of the claim, inside one forward pass and without rewriting the claim into sub-claims, and then aggregating those readings, recovers the fact-checkers' ordinal scale. The probe never sees a single gray-area label during training. Because it never sees those labels, it cannot learn the speaker, topic, and style shortcuts that make supervised fact-checking classifiers collapse out of domain. The same clause-level score also gives a graded hallucination measure for LLM answers, which lets the one mechanism be tested in both directions: from LLM factual errors to human misinformation and back.

Core Claim

A truth direction learned only from binary atomic statements and binary LLM-answer correctness, applied position by position within a compound claim, will match a probe trained directly on graded labels on in-domain ordinal agreement. It will clearly beat that probe under distribution shift, meaning new sources, new time periods, and new speakers. Where it fails, it will fail in a specific, predictable place: claims that are misleading through omitted context or cherry-picking while every clause is literally true. Graded veracity is therefore partly a compositional property that the binary geometry already supports, plus a remainder that is not about factual content at all. Measuring how large that remainder is counts as a result in its own right.

What Prior Work Establishes and Leaves Open

The motivating evidence comes from probing work. The Internal State of an LLM Knows When It's Lying and The Geometry of Truth show that truth is roughly linear in hidden states. HARP projects hidden states onto a low-variance "reasoning subspace" of the output layer and reports 92.8% AUROC on TriviaQA with almost no loss when transferred to NQ Open. CORAL finds that correctness signals sit in middle layers and are spread across many neurons rather than a few. All three use binary targets and pool over the whole sequence. We keep their middle-layer, distributed-direction recipe and change what is read (per-clause positions) and what is predicted (an ordinal degree).

The closest work is Latent Fact-Checking. It extracts a falsehood vector from contrastive "True"/"False" prompts, projects the final token of a bare claim onto it, and classifies LIAR claims in binary form (for example, Llama-3.2-1B rises from 0.47 to 0.70 accuracy). It also transfers from LIAR to FACTors. That paper uses LIAR's six-tier scale only as a binary task, reads one position, and fails on AVeriTeC, where truth depends on external evidence. We differ on the first two points. We treat the third as a scope constraint rather than something to fix: this proposal is about parametric knowledge, and we build in a knowledge-cutoff split to show where that scope ends.

The nuance literature shows why the ordinal scale matters and why the obvious text-level fix is risky. FACT5 reports that models confuse adjacent tiers. It then reaches large mean-squared-error reductions with a six-stage decompose-retrieve-aggregate pipeline, which is fragile because early decomposition errors carry through every later stage. Decomposition Dilemmas documents lost context, ambiguity, and changed meaning when claims are split into parts. FactLens identifies details fabricated during decomposition as the most damaging error mode. These papers constrain the design: we want the benefits of decomposition without an LLM rewriting the claim. That is why clauses are read inside the original forward pass, and split points come from a deterministic parser rather than a generator. X-Fact supplies the main warning about supervised learning. A claim-only mBERT reaches 38.2% macro F1 in-domain against 41.9% with evidence, and both fall to about 15–16% on out-of-domain sources. This is the shortcut behavior our label-free design is meant to avoid.

On the hallucination side, DiVA argues that binary verifiers are "blind" to degree. It builds a retrieval agent with a regression head scoring answer factuality continuously, and introduces FGVeriBench. RAGLens shows that internal sparse features beat the model's own chain-of-thought self-judgment, but it remains binary and specific to each model. We use FGVeriBench as the reverse-direction test: does a clause-level internal score rank LLM answers by degree of correctness without retrieval?

Research Program

We work with Llama-3.1-8B and Qwen-2.5-7B, the two models HARP studies, so the binary detectors are directly comparable. The truth direction is fit on binary data only: the true/false statement sets from the Geometry of Truth line of work, plus correct and incorrect TriviaQA answers generated by each model, following HARP's labeling. We train two versions per middle layer (roughly layers 14–22), choosing the layer on binary held-out data only. The first is a mean-difference direction in the style of Latent Fact-Checking. The second is a projection onto HARP's reasoning subspace followed by a linear read-out.

For clause reading, we pass each claim through the frozen model once. A dependency parser marks clause boundaries at coordinating and subordinating conjunctions, commas that end clauses, and sentence ends. We take the hidden state at the last token of each clause, so each reading covers the prefix up to that point. We then compute the model's incremental belief change: the projection at clause $k$ minus the projection at clause $k-1$. This separates what each clause contributes from the belief already built up by earlier clauses. Each clause gets a probability $s_k$ of being true. The claim's degree is a monotone aggregate $g(s_1,\dots,s_K)$. We test two versions. The first has no parameters: the mean of the $s_k$, mapped to the ordinal scale by quantiles. The second is a three-parameter monotone function of the minimum, the mean, and the fraction of clauses above 0.5, fitted on 200 graded claims. This small calibration set is the only contact with gray-area labels, and it is deliberately far smaller than what a supervised probe gets.

Two analyses turn the system into a scientific test. The first is a mechanism check on FactLens. That dataset has manually verified sub-claims, so we can ask whether each clause reading agrees with the verdict on its aligned sub-claim. If clause readings do not track sub-claim truth, a good ordinal score would be coincidental. The second is a residual analysis: we collect claims where the clause-level score says "true" but fact-checkers say "half true" or worse. We annotate a sample by the mechanism of deception (missing context, cherry-picked statistic, false causal link, satire, following the kinds of distinctions drawn in Fakeddit and Co-Creation of Misinformation Management Policies). We then test whether a second linear direction, fit on those residuals, generalizes across datasets. That test answers whether "misleading but literally true" has its own geometry.

Baselines

The naive reference points are a majority-class predictor and zero-shot prompting of the same 8B model to output a tier. On LIAR's six classes, the original text CNN reached 0.270 test accuracy (“Liar, Liar Pants on Fire”). We rerun the CNN only as a sanity anchor, not as a competitor.

The single-position graded projection reuses Latent Fact-Checking's final-token falsehood score, mapped to tiers with the same quantile and three-parameter calibration we use. This is the key ablation. It differs from our method only in reading one position instead of clauses, so any gain isolates the compositional reading. There is no reported result found on Lacuna for this method on six-class LIAR, because that paper reports binary accuracy only. Beating it means a statistically significant gain in quadratic-weighted kappa with the same direction, layer, and calibration data.

The strongest competitor on in-domain ordinal accuracy is a supervised ordinal probe. This is a two-layer network on HARP-projected, mean-pooled middle-layer states, trained with an ordinal (cumulative-link) loss on the full LIAR training split and on the X-FACT training split. It sees every graded label we withhold, so it is the fair test of whether withholding labels costs accuracy. Both probes get identical hidden states, layers, and a matched hyperparameter search of 30 trials. Our claim survives only if we come within one kappa point in-domain and beat it out of domain.

For external comparison on X-FACT, we report the published Attn-EA + Meta result of 41.9% macro F1 in-domain and roughly 15–16% on out-of-domain sources and unseen languages (X-Fact). Beating it means exceeding 16% macro F1 on the out-of-domain split without evidence or metadata. We restrict to X-FACT's English and high-resource-language subsets, where the model's knowledge is plausible, and state this restriction. On FACT5, FACT5's own decompose-retrieve pipeline is the evidence-using ceiling. The file reports only relative MSE reductions (71.9% for Mistral-7B at pass@3), so we rerun the released pipeline with our backbone to get absolute MSE. On FGVeriBench, DiVA reports 84.24% Precision@1 on NQ and 79.87% on Musique. Matching it without retrieval would be surprising. The meaningful bar is beating the reported FactScore figure of 77.64% on NQ and approaching DiVA at a small fraction of its latency.

Why not just…?

Why not just train a multi-class probe on the graded labels? This is the real threat. In-domain it may well win, because LIAR labels correlate with speaker and topic, and a supervised probe on hidden states can pick those up as easily as a text classifier. X-Fact's claim-only 38.2% F1 is the evidence that such shortcuts are strong. Our experiments settle the question with speaker-disjoint and source-disjoint splits, the X-FACT out-of-domain partition, and a date split. If the supervised probe loses little under these shifts, the label-free design has no advantage, and we would report that the gray-area signal is simply learnable.

Why not just decompose the claim in text and probe each sub-claim? It is a reasonable hybrid and we run it as an ablation, with LLM decomposition followed by the same truth direction on each atom. Decomposition Dilemmas and FactLens show that rewriting loses context and fabricates details, and that it hurts on simple claims. Our bet is that parser-based clause reading keeps the context the model already built up, while avoiding the generation step that introduces fabrications. If the text-decomposition hybrid wins on FACT5's complex statements, the lesson would be that clause boundaries are too coarse. We would then switch to reading at the atom boundaries a decomposer suggests, without substituting the decomposer's text.

Why not just prompt a strong model for a five-way rating? Frontier judges are strong but expensive: MiniCheck reports $107 versus $0.24 per 13,000 samples for GPT-4 against a small verifier (MiniCheck). Latent Fact-Checking shows that small models' prompted verdicts can fall below chance where their activations do not. We include prompted ratings from the same backbone as a baseline. We do not claim to beat a frontier judge with retrieval on post-cutoff claims, and the date split will show that boundary explicitly.

Proposed Experiments

The primary evaluation uses LIAR (six tiers, speaker-disjoint split), X-FACT's in-domain and out-of-domain partitions, and FACT5 (five tiers). The metrics are quadratic-weighted kappa, Spearman correlation with tier, mean absolute tier error, and macro F1. The reverse-direction test uses FGVeriBench with Precision@1, comparing our clause-level score against a pooled binary HARP score to see whether degree information helps rank LLM answers. The mechanism check uses FactLens clause-to-sub-claim agreement, measured by AUROC.

Three results would decide the proposal. First, if clause reading does not beat the single-position graded projection on FACT5 and LIAR, the compositional account is wrong and graded veracity is not assembled from parts. Second, the gap between our method and the supervised ordinal probe should flip sign from in-domain to out-of-domain. If it does not flip, the label-free design has no advantage. Third, on FACT5 split at each model's knowledge cutoff, clause reading should hold up on pre-cutoff claims and fall toward prompting on post-cutoff claims. This would replicate the AVeriTeC failure of Latent Fact-Checking along a controlled axis and confirm that the signal is parametric knowledge rather than style. The residual analysis then tells us whether the remaining "misleading but literally true" errors share one transferable direction. If they do, that is the second axis a real fact-checking probe needs.