Box-Valued Sentences for Directional Faithfulness Metrics

Most automatic metrics for judging generated text, from word-overlap scores like ROUGE to embedding scores like BERTScore, compare texts through a symmetric notion of similarity. That symmetry hides a distinction evaluation cares about. A summary that says "a dog bit the child" when the source says "a poodle bit the child" is vaguer but still true. A summary that says "a poodle bit the child" when the source only says "a dog" has invented a detail. Both pairs look almost equally similar to a cosine score and to n-gram overlap. We propose to represent each sentence as a region instead of a point: specifically a box, an axis-aligned hyperrectangle whose volume grows with how general the sentence is. Faithfulness then becomes a question of whether the source's region sits inside the claim's region. Informativeness becomes how much smaller the candidate's region is than the space of all meanings. Because regions can be intersected, a whole source document gets one region, the intersection of its sentences' regions. A claim that only follows from two source sentences together can then be checked without an LLM reading the whole document.

The second half of the proposal is about how such a metric should be judged. Standard meta-evaluation sets such as SummEval have been public for years. LLM judges may have seen both the texts and the human ratings, and contamination studies in this file show how much memorization can inflate scores. We therefore pair the metric with a freshly generated, directional test suite built from documents published after the judges' training cutoffs. Each item's correct verdict follows from the perturbation that produced it, not from a human label a model might have memorized.

Core Claim

Our claim is that a box-valued sentence encoder, trained on entailment and paraphrase data and scored by soft volume ratios, separates generalization (faithful) from specialization (hallucinated) edits, including edits with no shared words. Symmetric point metrics cannot do this, and a same-backbone directional entailment classifier does it less well once the evidence spans multiple source sentences. If a matched point-embedding entailment classifier works as well on multi-sentence cases, the geometry is not earning its keep, and the experiments are designed so that this outcome is visible.

Concretely, let $B_s$ be the box assigned to sentence $s$, and let $B_R = \bigcap_j B_{r_j}$ be the intersection of the boxes of the reference sentences $r_j$. We score a candidate claim $c$ by

$$F(c \mid R) = \frac{\mathrm{Vol}(B_R \cap B_c)}{\mathrm{Vol}(B_R)},$$

which is near 1 when everything the reference asserts implies $c$. Volumes use the Gumbel soft-volume relaxation, so the score is differentiable and nonzero even for boxes that do not overlap. The summary-level faithfulness score is the minimum or a soft-minimum of $F$ over candidate sentences. Informativeness is $-\log \mathrm{Vol}(B_C)$, where $B_C$ intersects the candidate's boxes. A summary can therefore be faithful but empty (large box), or specific but unfaithful (small box outside the reference), and the metric reports these separately.

Relation to Prior Work

Three papers supply the machinery and motivate the geometry. Word2Box learns static word boxes that support intersection and set difference. The Box Embeddings library provides the soft intersection and volume operators we reuse, and it reports that boxes on a RoBERTa encoder reach 78% accuracy on MNLI. ProtoBox makes boxes contextual for hypernym identification. None of them uses box volume as a graded score for comparing generated text with a source, and none composes a document by intersecting sentence boxes. The MNLI result is therefore our starting point, not our contribution. Chen and Gimpel's probabilistic sentence model is the closest conceptual precedent. It represents sentences as Gaussians whose entropy tracks specificity, and in 75.8% of SNLI pairs the premise has lower entropy than the hypothesis. Its matching score is symmetric (expected inner product) and was never used as a metric. We keep its insight that spread measures generality and replace the symmetric score with a directional containment score.

The evaluation literature supplies the gap. SummEval found that several neural metrics correlate with factual consistency no better than ROUGE-3 or METEOR. That is a warning that dense point similarity can blur swapped entities, and it constrains us: the box metric must beat n-gram metrics on consistency specifically, not only on paraphrase robustness. Zeidler et al.'s meaning-oriented CheckList gives 939 SICK/STS pairs labeled by phenomenon, including hyponymy, negation and role switch, and shows graph metrics handle some of these through discrete meaning-graph parses rather than geometry. HD-Eval shows that splitting quality into sub-criteria helps LLM judges. Its hierarchy lives in prompts, while ours lives in the representation. NSnet separates lexical gaps from knowledge gaps on SciTail by decomposing hypotheses into sub-facts. Our intersection over reference sentences is a continuous analogue of that decomposition on the source side.

The contamination literature constrains how we evaluate. PAC estimated a 91.4% contamination rate for GPT-3 davinci-002 on GSM8K. DyVal showed that freshly generated reasoning instances can collapse scores that looked strong on static sets. DyVal is confined to formal graphs, though. Our suite is a natural-language counterpart: it generates new items whose ground-truth direction is fixed by construction.

Research Program

We would build three things.

The first is the encoder. A pretrained transformer encoder is followed by a head that outputs a box center and positive offset per sentence. It is trained with a containment loss on SNLI/MNLI entailment pairs, where premise should sit inside hypothesis and contradictions should be disjoint. It also gets a mutual-containment loss on ParaNMT paraphrases, so that different wordings of the same meaning yield near-identical boxes; this is how the lexical gap is handled. Finally, we add hypernym-substitution pairs derived from WordNet, so that a sentence with a more general word receives a strictly larger box.

The second is the metric. It computes $F$ and informativeness as defined above. We also study how many reference sentences can be intersected before soft volumes become numerically unstable, since empty intersections are the known failure mode of hard boxes.

The third is the fresh directional suite. Source sentences come from documents timestamped after the evaluated judges' cutoffs. Each is edited in one of six controlled ways: synonym or paraphrase rewrite (faithful, low word overlap), hypernym generalization (faithful), hyponym specialization (unsupported), negation (contradiction), entity or role swap (contradiction), and a merge of two source sentences into one claim (faithful but supported only jointly). Paraphrases are produced by a model and filtered by human spot-checks. Because the direction of each edit is known, labels come from construction rather than annotation. A small human-verified subset measures label noise.

Baselines

The naive reference points are ROUGE-L, ROUGE-3 and METEOR, the last two because SummEval found them unexpectedly competitive on consistency. We also run BERTScore precision and recall, computed with the same backbone encoder as ours so that differences come from the scoring geometry rather than the pretrained model.

The strongest existing learned competitor is a directional entailment classifier. It uses the same backbone, is fine-tuned on exactly the same entailment, paraphrase and hypernym data with the same tuning budget, and is applied per candidate sentence as the maximum entailment probability over reference sentences, plus a variant that feeds the whole reference as the premise. This is the baseline most likely to match us. We know of no reported SummEval consistency correlation for it on Lacuna. As a sanity anchor we reproduce the library's 78% box-on-RoBERTa MNLI accuracy before comparing.

The strongest judge baseline is HD-Eval with GPT-4. The file reports it as roughly a 15% relative Pearson improvement over plain GPT-4 prompting on SummEval, but gives no absolute correlation, so we will rerun it ourselves on both SummEval and the fresh suite. For SciTail the anchor is NSnet's 77.9% accuracy against 74.3% for its neural base.

The key ablation swaps the box head for a point head of equal parameter count, trained with the same losses rewritten as asymmetric score functions. This isolates whether region geometry, rather than the directional training data, produces any gain. A second ablation replaces reference-box intersection with a per-sentence maximum, isolating composition.

Beating the matched classifier means higher accuracy on the specialization-versus-generalization split and on merged-claim items of the fresh suite, with non-inferior Kendall correlation on SummEval consistency. Everything is judged with bootstrap confidence intervals.

Why not just…?

Why not just use a fine-tuned entailment classifier, which is already directional? This is the real threat, and on single-sentence pairs it may tie us. The box metric's distinct claim is compositional. A classifier either sees one reference sentence at a time, missing claims that need two, or reads the whole document, where long inputs degrade it. The merged-claim slice of the fresh suite and the composition ablation settle this directly. If the classifier with whole-document input matches us there, we will report that boxes add interpretability but no accuracy.

Why not just ask a frontier LLM to judge faithfulness? It probably handles hypernym direction well. However, DyVal found 33.8% of GPT-4's failures were unsubstantiated responses and a 13.6% drop from merely reordering descriptions, and SummEval-based comparisons may be inflated by exposure to the benchmark. We run the judge on both static and fresh data. The gap between its two scores, compared with ours, is itself a finding about how much static meta-evaluation flatters LLM judges. Cost per scored summary is reported alongside.

Why not just use BERTScore precision versus recall for asymmetry? Its asymmetry comes from token matching, not from meaning. "Poodle" matches "dog" with similar strength in both directions, so specialization and generalization edits should score alike. The hyponymy slice of Zeidler et al.'s CheckList and our generalization/specialization slice test exactly this.

Decisive Checks

The first check is the direction test on the fresh suite. The box metric must assign higher $F$ to generalization than to specialization edits in a clear majority of matched pairs; WLO's 75.8% SNLI directionality is the bar to clear. Every symmetric baseline should sit near chance on this contrast by construction.

The second check is the matched-classifier comparison on merged claims. The third is correlation with SummEval expert consistency ratings, using expert rather than crowd ratings because expert agreement is higher (alpha 0.71 versus 0.49). This must at least match ROUGE-3; otherwise the SummEval warning about dense metrics applies to us too.

The last check is ranking stability. We compute each metric's system-level ranking on static SummEval and on fresh summaries of post-cutoff articles, and test whether LLM judges lose more agreement than the box metric when moving to fresh data. A negative result on the first two checks would falsify the core claim. A negative result on the last would only mean contamination is not what separates these metrics.