Focal Probes for Factuality Metrics That Survive Domain Shift

Automatic checkers that decide whether a summary is faithful to its source document work well on news. They break elsewhere. A classifier trained on CNN/DailyMail falls from roughly 73% to 53% accuracy on the more abstractive XSum, and question-answering metrics tuned on news show near-zero agreement with humans on biomedical text. This note argues that the collapse mixes two different failures. The first is that the base rate of errors changes across domains: 43% of CNN/DM summaries contain a factual error versus 92% on XSum. The second is that the checker has learned shortcuts, such as treating word overlap with the source as evidence of faithfulness, and those shortcuts stop working out of domain. The two failures need different fixes. Folding them into one accuracy number has made it hard to tell which remedies actually help.

We propose a cheap evaluator with three parts. Features come from a natural-language-inference model that is frozen, meaning never updated; such a model scores whether one sentence logically follows from another. A small classifier sits on top of those features. Its training and calibration are split across three stages. The decision direction, which determines how summaries are ranked, is learned with focal loss, a variant of cross-entropy that shrinks the contribution of examples the model already gets right. The probability scale is then refit with ordinary cross-entropy. Finally, the error base rate is corrected separately for each target domain. The core bet is that, on a frozen representation, focal loss earns its keep by down-weighting easy shortcut examples, not by handling class imbalance. The second stage is needed because focal loss distorts probabilities, and that distortion breaks the base-rate correction.

Core Claim

On frozen features, a standard (unweighted) focal loss improves the threshold-free ranking quality of a factuality probe on unseen domains. Class-reweighted cross-entropy does not, because reweighting by class changes only the intercept of a linear head, not its direction. The ranking gain from focal loss should be concentrated in domains where lexical overlap stops predicting faithfulness. Focal loss also leaves the head under-confident. If its raw probabilities are used for base-rate correction, the gain in ranking is cancelled by a loss in thresholded accuracy, and recalibrating the head separately recovers it. If ranking quality (area under the ROC curve) does not improve under focal loss while class reweighting matches it, the claim is false.

Positioning

Two lines of prior work motivate the proposal. The first shows that frozen features with small heads transfer better than fine-tuned evaluators. IDK trains an L1-regularized logistic regression on frozen BERT features for dialogue relevance, with one fixed negative example. It cuts the ratio between its best and worst cross-dataset correlation from 6.2 for a fine-tuned model to 3.9. We keep its frozen-head design but study the loss rather than the negative-sampling heuristic, and move from dialogue to factuality. SummaC supplies our feature extractor. It scores every document-sentence and summary-sentence pair with a frozen NLI model and aggregates the resulting matrix with a small learned convolution, reaching 74.4% balanced accuracy. It trains that aggregator with plain cross-entropy and tunes thresholds per dataset. That per-dataset tuning hides exactly the base-rate shift we want to measure.

The second line diagnoses the shortcut. FactCC trains on balanced, rule-generated errors and collapses on XSum. FRANK attributes this to overfitting on extractive errors and failure on facts not found in the article. Both papers motivate the hypothesis that easy training examples carry surface cues.

Focal loss itself comes from Lin et al.. Applications such as ASRtrans and the primate-call classifier of Müller et al. use it in end-to-end training, and neither studies frozen heads or tests transfer.

The calibration literature constrains the design. Dual Focal Loss shows that fixed-$\gamma$ focal loss produces under-confidence. AdaFocal shows that the miscalibration varies across confidence ranges. Functional Properties of the Focal-Entropy proves an "over-suppression regime" under extreme imbalance, in which small probabilities are pushed down further. These results are why our design does not trust the focal head's probabilities. We know of no work that combines frozen probes, focal loss, and cross-domain metric evaluation. More importantly, none separates the ranking and base-rate components of transfer.

Research Program

The features are SummaC-style entailment histograms, concatenated with a pooled embedding of the summary under the frozen NLI encoder. The head is a linear model or a one-hidden-layer network. Training uses the CNN/DM-derived portions of the SummaC benchmark: CoGenSumm, FactCC, SummEval, and the CNN half of FRANK. Each head is trained in three stages.

The first stage learns the head's weights by minimizing focal loss $-(1-p_t)^\gamma \log p_t$. Here $p_t$ is the head's probability for the correct label and $\gamma$ sets how strongly confident examples are down-weighted.

The second stage freezes the weights and refits only a temperature and bias with cross-entropy on held-out source data. This restores probabilities that can be interpreted as likelihoods.

The third stage handles each target domain. We estimate the error base rate from unlabeled target summaries by expectation-maximization over the calibrated posteriors, then shift the posteriors by the standard prior-ratio correction:

$$p_T(y\mid x) \propto p_S(y\mid x)\,\frac{\pi_T(y)}{\pi_S(y)}$$

In this expression, $p_S$ is the calibrated source posterior, $\pi_S(y)$ is the source base rate of label $y$, and $\pi_T(y)$ is the estimated target base rate. The correction is only valid if $p_S$ is calibrated, which is why the second stage exists.

The analysis has two parts. Every reported transfer number is decomposed into a ranking component, measured as area under the ROC curve (AUC), and a thresholding component. The thresholding component is the gap between balanced accuracy at a fixed threshold and balanced accuracy at the best threshold chosen with target labels. To test the shortcut mechanism, we measure, in each domain, the partial correlation between the head's score and ROUGE overlap with the source after controlling for the human label. We also record which training examples focal loss down-weights most, and check whether they are disproportionately high-overlap extractive cases.

Baselines

The naive reference is a cross-entropy logistic probe on the same frozen features with a 0.5 threshold. It shows how much transfer error comes from doing nothing.

The direct competitor for the loss question is class-balanced cross-entropy on the same features, with the class weight tuned over the same grid size as $\gamma$. It is the strongest competitor because it is what a practitioner would try first for skewed labels. Beating it means higher target AUC with overlapping-free bootstrap confidence intervals, not just higher thresholded accuracy.

A hard-example-mining control trains cross-entropy after discarding the easiest k% of source examples under a first-pass probe, with k tuned like $\gamma$. This ablation isolates whether focal loss's smooth reweighting matters or whether simply removing easy examples suffices.

Dual Focal Loss and AdaFocal heads, trained with identical features and tuning budget, test whether a calibration-aware focal variant makes our second stage unnecessary.

Among existing systems, we use SummaC-Conv, with its reported 74.4% balanced accuracy on the SummaC benchmark, as the frozen-feature reference. The strongest published system on this benchmark is QAFactEval-NLI, which reports 79.5 balanced accuracy on the SummaC validation set (QAFactEval), versus 77.5 for plain QAFactEval. We rerun both on our leave-domain-out splits, since their published thresholds were tuned in-domain. Our probe beats QAFactEval-NLI if it matches or exceeds it at fixed threshold on the held-out domains while costing a small fraction of the compute.

On SummEdits, GPT-4 reports 82.4% balanced accuracy against a human 90.9% (SummEdits). This is the ceiling we compare against there, not a target we expect to beat.

Why not just…?

Why not just tune the decision threshold on each target domain? This is the most serious threat. If base-rate shift explained most of the collapse from 72.88% to 53.05% that FactCC suffers on XSum, per-domain thresholds alone would recover most of it, and focal loss would add nothing. Our decomposition tests this directly. Threshold tuning cannot change AUC, so any focal-loss gain in AUC is something threshold tuning cannot deliver. If the AUC gains are negligible, we will report that the fix is base-rate correction alone. Our third stage also needs no target labels, whereas threshold tuning does.

Why not just reweight classes in cross-entropy? For a linear head on fixed features, class reweighting mainly moves the intercept. It cannot reorder examples, so it should help thresholded accuracy but not ranking. ASRtrans found focal loss necessary under heavy skew, but in end-to-end training and without this comparison. The class-balanced baseline, tuned on an equal budget, settles the question.

Why not just fine-tune the NLI model end-to-end? IDK shows that fine-tuned evaluators overfit across datasets. Mayfield and Black show that frozen heads come within 0.01–0.04 QWK of fine-tuning on essay scoring at about 1% of the training time. We still include a fine-tuned RoBERTa-MNLI trained with cross-entropy and with focal loss as an upper reference for in-domain accuracy. If it transfers better in AUC, the frozen-head premise is weakened, and we will say so.

Why not just ask a large language model to judge? SummEdits reports most LLMs between 50% and 70%, and FAREBIO finds LLM judges overconfident on biomedical hallucinations. GPT-4's 82.4% is strong, however, so we position the probe as a cheap, auditable alternative and report GPT-4 only where it already has published numbers.

Proposed Experiments

The evaluation is leave-domain-out. Probes are trained on the CNN/DM-derived SummaC subsets and tested on XSumFaith, the XSum half of FRANK, and Polytope. Further tests use all ten SummEdits domains, where label skew differs by domain, and FAREBIO, where sentence-level error rates of 3–17% put us near the over-suppression regime. Metrics are AUC, balanced accuracy at fixed and at oracle thresholds, expected calibration error, and the overlap partial correlation, all with bootstrap intervals over five seeds.

The decisive checks are as follows. First, focal heads must beat class-balanced cross-entropy in target AUC. Second, the gain must grow with how much a domain's overlap-faithfulness relationship departs from the source domain's. Third, base-rate correction applied to uncalibrated focal probabilities must underperform the version with the second-stage recalibration, especially on FAREBIO, which would confirm that calibration is a prerequisite for the correction. Fourth, the hard-mining control tells us whether a simpler mechanism already explains the effect. If the first check fails, we abandon the focal-loss claim and report the base-rate decomposition as the main finding.