A contract-review model asked whether "the agreement lets the recipient share confidential information with employees" can be wrong for two different reasons. The contract may say nothing relevant, so the model is guessing in a vacuum. Or the contract may say yes in one clause and "except as provided in Section 9" in another, so the evidence actually disagrees. A human reviewer needs different things in each case: in the first, confirmation that the topic is absent; in the second, a pointer to the two clashing clauses. Today's systems hand the reviewer one confidence number, and that number merges these cases.
This note proposes a small, trainable fusion layer that sits on top of an existing clause-level entailment model and keeps the two cases apart. Each clause casts a partial vote for "entailed" or "contradicted," and the unspent part of its vote means "this clause says nothing." The votes are pooled with one tunable exponent that slides between "any single clause settles it" and "clauses are averaged." The pooled result separates evidence for, evidence against, their clash, and the leftover silence. The system then abstains with a statistically guaranteed error rate, and it reports why it abstained. The central claim is that, in document-level entailment, separating clash from silence and letting a single exception clause count at full strength improves abstention exactly where current systems fail worst: contradictions created by exceptions.
Core Claim
Textual entailment asks whether a hypothesis follows from a premise. ContractNLI makes this document-sized. It pairs 607 non-disclosure agreements with 17 fixed hypotheses and labels each pair Entailment, Contradiction, or NotMentioned (ContractNLI). Its authors report that their span-based model reaches 88.5% accuracy but only about 0.36 F1 on contradictions, against about 0.85 on entailment. They trace much of the gap to evidence that is scattered, which happens in about 30% of cases, and to "negation by exception."
The observation that makes this tractable is that ContractNLI's label set already has the structure of Dempster–Shafer evidence theory. That theory spreads belief over sets of answers rather than single answers, so belief left on "either answer" means "no information." Take the set of possible answers to be just {Entailed, Contradicted}. Then NotMentioned is not a third answer. It is the belief that stays on the whole set. Because the set has only two elements, the usual exponential cost of this theory disappears, and every quantity can be written in closed form.
For clause $i$ of a document, a fine-tuned clause-level model outputs support $e_i$ for entailment, support $c_i$ for contradiction, and a relevance weight $r_i$, all in $[0,1]$. We pool each kind of support with a relevance-weighted power mean, the operator that Power mean pyramid scores used to tune summary scoring between strict and lenient:
$$s_E = \Big(\tfrac{\sum_i r_i e_i^{\alpha_E}}{\sum_i r_i}\Big)^{1/\alpha_E}, \qquad s_C = \Big(\tfrac{\sum_i r_i c_i^{\alpha_C}}{\sum_i r_i}\Big)^{1/\alpha_C}.$$
The exponents $\alpha_E$ and $\alpha_C$ are learned per hypothesis type. A large $\alpha_C$ approaches the maximum, so one strong exception clause registers at full strength instead of being outvoted by many permissive clauses.
We then treat $s_E$ and $s_C$ as two simple pieces of evidence and combine them without normalization:
$$m(E)=s_E(1-s_C),\quad m(C)=s_C(1-s_E),\quad m(\Theta)=(1-s_E)(1-s_C),\quad K=s_E s_C.$$
Here $m(\Theta)$ is silence, the NotMentioned belief, and $K$ is the clash between the two kinds of support.
Three design choices follow from the literature. The power mean is symmetric in the clauses, so the order in which clauses are read does not matter. This sidesteps the associativity failures that Reducing t-norms and augmenting t-conorms proves for flexible operators, because we pool all clauses at once rather than combining them one pair at a time. Leaving $K$ unnormalized avoids the breakdown of Dempster's rule under near-total conflict, which Tao and Xiao work around with entropy-ordered weighting. Finally, $K$ and $m(\Theta)$ become separate triggers for abstention.
Relation to Prior Work
Four papers motivate the proposal by exposing the gap. Fan et al. show in image recognition that splitting "confusion" from "ignorance" matters. Their confusion measure reaches AUROC 90.0 versus 57.3 for softmax at identifying misclassified CIFAR-100 samples. They work on single images with fixed rules, not on pooled textual evidence. Ruan et al. fuse clinical text and tables with Dempster's rule, but their model never abstains and treats each note as one block. Shiraishi et al. build an explicit "I don't know" belief into fuzzy rule classifiers, then convert it back into a forced choice. We keep that belief and act on it. Uncertain NLI shows that the Neutral label spans nearly the whole probability range, yet it models uncertainty as a single scalar, which cannot tell a clash from silence.
Two lines of work supply methods we reuse. The first is selection with guaranteed risk (Geifman and El-Yaniv): from a held-out set and a binomial bound, it picks an abstention threshold whose true error stays below a target with high probability. The second is the clause-level segmentation of Span NLI BERT (ContractNLI).
On the selective-prediction side, Varshney et al. train a secondary calibrator for sentence-level NLI. Attribute or Abstain ties abstention to failed evidence citation on long documents, including ContractNLI. Neither gives a risk guarantee or separates the reasons for abstaining. They are our main competitors, not our building blocks.
Research Program
We fine-tune a DeBERTa clause-level model on ContractNLI clause–hypothesis pairs. The labels come from gold evidence spans: evidence clauses inherit the document label, and all other clauses are "silent." The model outputs $(e_i, c_i, r_i)$.
The fusion layer learns $\alpha_E$ and $\alpha_C$ per hypothesis by gradient descent on a document-level loss. That loss rewards belief on the gold label, including putting $m(\Theta)$ on NotMentioned. Everything else is frozen or fine-tuned jointly. We compare the two regimes.
The decision rule predicts the label with the largest of $m(E)$, $m(C)$, $m(\Theta)$. Abstention uses two scores, $K$ and $1-\max(m)$. Thresholds are chosen on a calibration split. To keep the risk guarantee valid, we search a fixed, pre-registered grid of threshold pairs and apply a union bound over the grid, so the guarantee holds at level $1-\delta$ after the search. Each abstention is tagged "conflict" if $K$ triggered it and "insufficient evidence" otherwise.
We also build two diagnostic test sets from ContractNLI. For the exception set, we take entailed documents and insert a real exception clause copied from a contradicted document for the same hypothesis. The correct behaviour is that $K$ rises and the label flips or the system abstains as "conflict." For the removal set, we delete the gold evidence spans. The correct behaviour is that $m(\Theta)$ rises and the prediction becomes NotMentioned or an "insufficient evidence" abstention.
As a secondary test of generality under shift, we apply the same two-element set to sentence-level NLI: train on SNLI, test on MNLI and the NLI stress test, with Neutral read as silence. This follows the evaluation setup of Varshney et al..
Baselines
The naive reference is MaxProb on Span NLI BERT: the published architecture, retrained on our splits, abstaining by maximum softmax probability. ContractNLI reports 88.5% accuracy and roughly 0.36 contradiction F1 for this model without abstention. I found no reported risk–coverage result on Lacuna for any selective method on ContractNLI, so our MaxProb run defines the reference curve.
The strongest existing selective method is the calibrator of Varshney et al.. It is a random-forest regressor over model probabilities, training-dynamics difficulty scores, and shallow features. It is the most direct competitor because it was designed for NLI abstention. On SNLI/MNLI it reports AURCC improvements over MaxProb of 15.81% in-domain and 5.64% out-of-domain; that metric is the area under the curve of error rate versus fraction answered. We reimplement it on the same backbone, splits, and tuning budget.
The key ablation is the same random forest given our features: $s_E$, $s_C$, $K$, $m(\Theta)$, and the per-clause maxima and counts. This separates two possible sources of gain: the evidence features themselves, and the structured combination plus the risk-guaranteed, reason-tagged selection.
Further ablations each remove one ingredient. One fixes $\alpha=1$ (plain averaging). One replaces power-mean pooling with normalized Dempster combination across clauses. One collapses the two abstention triggers into a single score. We also include MC-dropout variance over 10 passes and a scalar regression head in the style of Uncertain NLI.
Winning means three things. First, AURCC lower than both the Varshney calibrator and the same-features random forest, with non-overlapping bootstrap intervals across 5 seeds. Second, higher coverage at a guaranteed 5% risk. Third, contradiction F1 on the answered cases that clearly exceeds MaxProb's at matched coverage.
Why not just…?
Why not just train a calibrator on hand-built clash and silence features? This is the real threat. If the random forest given $K$ and $m(\Theta)$ matches our system on AURCC, then the useful part is the features, not the evidence-theoretic combination, and we will report that honestly. Our system would still keep two properties the calibrator lacks: a closed-form rule that handles any number of clauses without retraining, and abstentions that carry a named reason. The exception and removal test sets check whether the named reasons are actually right, which a calibrator's single score cannot be tested on.
Why not just train a better end-to-end document model, with learned attention pooling and contradiction upweighting? Stronger pooling may raise contradiction F1. But a single softmax output still merges clash and silence, which is the failure Fan et al. document. We include this model as a backbone variant, putting MaxProb on top of it, and also apply our fusion layer over its clause outputs. That tests whether the gains add to a stronger model rather than replace it.
Why not just let an LLM cite evidence and abstain when citation fails? Attribute or Abstain finds that fine-tuned Flan-T5-XL gives the best response and evidence quality on its long-document benchmark. Its abstention, however, is a heuristic without a risk bound. The file gives no ContractNLI-specific score from that work. We run fine-tuned Flan-T5-XL with the citation approach on our splits, using the same calibration-split threshold procedure. We expect citation failure to detect silence well but miss clash, because an exception clause is itself citable. The exception test set measures exactly this.
Decisive Checks
The proposal lives or dies on four results. On ContractNLI, it must beat the same-features random forest on AURCC and on coverage at 5% guaranteed risk. On the exception set, $K$ must separate inserted-exception documents from untouched entailed documents with AUROC well above MaxProb's. On the removal set, $m(\Theta)$ must do the same for evidence-removed documents. Finally, the learned $\alpha_C$ must settle clearly above $\alpha_E$ on hypotheses that are prone to exceptions, which would show that the tunable pooling captures "one exception overrides many permissions."
If the diagnostic sets pass but AURCC ties the calibrator, the contribution narrows to interpretable, risk-guaranteed, reason-tagged abstention. If the diagnostics fail, the clash-versus-silence distinction does not transfer to text at clause granularity.
References
- ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts
- Power mean pyramid scores for summarization evaluation
- Reducing t-norms and augmenting t-conorms
- Combine Conflicting Evidence Based on the Belief Entropy and IOWA Operator
- Flexible Visual Recognition by Evidential Modeling of Confusion and Ignorance
- Towards Accurate and Reliable ICU Outcome Prediction: A Multimodal Learning Framework Based on Belief Function Theory using Structured EHRs and Free-Text Notes
- A Class Inference Scheme with Dempster–Shafer Theory for Learning Fuzzy-Classifier Systems
- Uncertain Natural Language Inference
- Selective Classification for Deep Neural Networks
- Towards Improving Selective Prediction Ability of NLP Systems
- Attribute or Abstain: Large Language Models as Long Document Assistants