Evidence Spans as Graded Entailment Support: Tagging What Makes a Lay Story a Legal Case

When a person writes to a legal aid service ("my son took over my bank card after my fall and won't let me see statements"), a lawyer can say both that this is an Elder Law matter and which words justify that call. Machines are poor at the second part. On a dataset of 35,000 such descriptions, each highlighted by five practicing lawyers, the best trained tagger reaches about 57% word-level F1 against a human ceiling of about 94%, and only 19.0% exact-span F1 (To Aggregate or Not to Aggregate). Two things make this hard. First, the client's words ("took over my bank card") share almost no vocabulary with the legal category they point to ("financial abuse of an older person"). This is a lexical gap. Second, the five lawyers often disagree about where the evidence begins and ends.

This note proposes that the right definition of an evidence span is a stretch of text that supports the claim "this story describes category $c$", in the sense used by natural language inference, the task of deciding whether one text logically supports another. It also proposes that lawyer disagreement is the graded version of the same quantity. Concretely, we build a token tagger that reads a written description of the legal category next to the client's text. It starts from a model already trained for inference and learns from the fraction of lawyers who marked each token, not from a majority vote. It is also trained so that deleting the predicted span should reduce the inference model's confidence that the category applies, and keeping only the span should preserve it. The testable bet has two parts. Tokens more lawyers agree on should be the ones whose removal hurts entailment most. And tying tagging to entailment should close part of the human–model gap, including on legal categories never seen in training.

Core Claim

Existing work splits along a fault line this proposal crosses. Entailment-based classifiers turn labels into sentences and score them with an inference model. They bridge vocabulary gaps between text and label names, but they always receive the span from outside. LITE ranks hypotheses such as "[entity] is a [type]" over given mentions of about 10,000 types. It reaches 50.6 F1 on ultra-fine typing and 32.9 F1 with 40% of types unseen, against 10.8 for the prior method (LITE). SETYPE does the same for software and security jargon with context templates (SETYPE). Neither finds span boundaries.

The span taggers go the other way. They assign BIO tags, a per-word scheme marking the Beginning of a span, Inside it, or Outside any span. The tags are arbitrary class indices with no meaning attached. That holds for the Justice Connect taggers (To Aggregate or Not to Aggregate) and for the winning toxic-span ensemble (HITSZ-HLT).

The Justice Connect study also leaves a dilemma about disagreement. Majority voting beat repeated labeling, which trains on each lawyer's highlight as a separate example: 18.2% vs. 11.8% span F1 for BERT. That seems to say disagreement is noise. Work on subjective labeling instead finds disagreement structured and informative (Reconsidering Annotator Disagreement about Racist Language; iLab at Le-Wi-Di).

Our claim resolves the dilemma. Repeated labeling fails because it feeds the tagger contradictory hard sequences. Majority voting throws away the graded signal. A tagger trained on per-token agreement distributions, and anchored to an entailment objective that gives "evidence" a meaning independent of any single annotator, should beat both. The closest structural precedent, Meta4XNLI, annotates metaphor tokens and inference pairs in one corpus but treats detection and inference as separate tasks with consensus labels (Meta4XNLI). We couple them in one model and use disagreement directly.

Research Program

The model is a cross-encoder of the kind used for inference, initialized from an inference-trained DeBERTa. Its input is a category hypothesis $h_c$ (a one-sentence plain-language description of legal category $c$) paired with the client text $x$. A token head over the positions of $x$ outputs BIO tags, decoded with a conditional random field, a layer that scores whole tag sequences so tags follow valid transitions. Because the category enters as text rather than as an index, a new category needs only a new description.

The training targets come from the five lawyer highlights. Each highlight becomes a BIO sequence. At each token $t$ we take the empirical distribution $q_t$ over {B, I, O} across the lawyers. The tagger is trained with cross-entropy between $q_t$ and the model's per-token marginal $p_t$. This keeps the boundary uncertainty that majority voting discards and avoids the contradictory hard targets of repeated labeling. The choice of soft targets follows the finding that soft pseudo-label distributions beat hard ones in sequence labeling (Risk Minimization for Zero-shot Sequence Labeling). Here the soft distributions come from real experts rather than noisy source models.

The entailment tie is a second loss term. We use a frozen copy of the inference model's entailment probability $E(\cdot, h_c)$ and soft-mask the token embeddings of $x$ by the predicted in-span probability $s_t = 1 - p_t(O)$:

$$ \mathcal{L} = \sum_t \mathrm{CE}(q_t, p_t) + \lambda \Big[ \max\big(0,\, E(x \odot (1-s), h_c) - E(x, h_c) + m\big) + \max\big(0,\, E(x, h_c) - E(x \odot s, h_c)\big) \Big] $$

Here $x \odot s$ keeps only predicted evidence and $x \odot (1-s)$ removes it. The first bracket asks that removing the evidence lower entailment by at least a margin $m$. The second asks that the evidence alone keep entailment roughly as high as the full text. The weight $\lambda$ balances the two losses, and a small sparsity penalty on $s$ is added in practice to rule out the trivial solution of marking everything. This loss is what lets the model tag categories it has never seen annotated spans for. Without it, the hypothesis-conditioned tagger is just a better-initialized supervised tagger.

The analysis side tests the premise directly, independent of any leaderboard gain. For every annotated token we measure how much deleting it (and its surrounding annotated span) lowers the frozen inference model's entailment probability. We then correlate that drop with the fraction of lawyers who marked it. A clear positive correlation supports the view that lawyer disagreement tracks graded evidential strength. A null result means entailment and legal evidence are different notions on this data, and the entailment term should be dropped. We also measure template sensitivity, a known weakness of entailment classifiers (Issues with Entailment-based Zero-shot Text Classification), by training and testing with several paraphrased category descriptions.

Baselines

The naive reference points are two taggers. One highlights every token of the description, which gives the word-F1 floor for a task where evidence is often a large fraction of short texts. The other highlights tokens matching a keyword lexicon built from category names expanded with WordNet synonyms. Both cost nothing, and the lexicon one tests whether surface matching bridges any of the lay–legal gap. The expanded toxic-word dictionary in HITSZ-HLT raised recall but hurt precision, and we expect the same.

The strongest existing method is the DeBERTaV3 + BiLSTM + CRF tagger trained on majority-vote tags from To Aggregate or Not to Aggregate. It reports 19.0% span F1, and the best model there reaches about 57% word F1. The repeated-labeling variant scores 18.2% span F1 for DeBERTaV3 and 11.8% for BERT. We reimplement it on the same splits with the same encoder size and the same hyperparameter search budget as our model, confirm we reproduce roughly 19.0 span F1 before comparing, and report both the reproduced and published figures. Beating it means a span-F1 improvement that holds across at least three seeds with non-overlapping bootstrap intervals, together with no loss in word F1.

Three ablations isolate the key ingredients, all at equal budget. The first is a soft-label tagger with no hypothesis, which tests the disagreement component alone. The second is a category-conditioned tagger in which the category enters as a learned embedding rather than a description, which tests whether verbalization matters. The third is our full model initialized from plain DeBERTaV3 rather than the inference-trained checkpoint, and with $\lambda = 0$, which tests whether the entailment grounding matters beyond extra inputs.

For held-out categories, the comparison is a zero-shot rationale extractor built on an inference classifier using sharpened soft attention (Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers). It is the most direct existing way to get tokens from a sentence-level judgment. Beating it on unseen categories is the zero-shot success criterion.

We also include a prompted instruction-tuned LLM asked to quote evidence for the given category, run with a fixed prompt tuned on development data only. No reported result found on Lacuna for LLM prompting on Justice Connect.

Why not just…?

Why not just keep majority-vote DeBERTa and train longer or bigger? It is the strongest published system, and scaling is cheap. But a 19.0% span F1 against a 94% human word ceiling suggests the model is missing the link between lay phrasing and legal categories, not just capacity. The category-embedding ablation measures exactly how much of our gain survives without verbalized, inference-pretrained hypotheses.

Why not just train on soft agreement labels and skip entailment entirely? This is the real threat. Soft targets alone may explain most of any in-category gain, given the evidence from Risk Minimization for Zero-shot Sequence Labeling. We therefore treat the in-category result as a test of the soft labels and the held-out-category result as the test of entailment. A soft-label tagger without a hypothesis cannot tag a category it never saw. If the full model matches it in-category and shows no zero-shot advantage over the attention-rationale baseline, we will report that the entailment coupling adds nothing.

Why not just prompt a large generative model to quote the evidence? On specialized vocabulary, prompted GPT-3.5 Turbo scored 50.79 Micro-F1 against 66.15 for an entailment-trained encoder in SETYPE. Meta4XNLI found encoders beat decoders on token-level metaphor detection. That evidence is about typing and metaphor, not legal evidence, so we run the prompted baseline rather than assume its outcome. LLMs also give no calibrated per-token distribution to compare against lawyer agreement.

Why not just classify with an inference model and read off attention or erasure scores? Raw attention and LIME spread credit over irrelevant tokens and give poor boundaries (Zero-shot Sequence Labeling for Transformer-based Sentence Classifiers). Boundary-only span models also struggled on multi-span texts in HITSZ-HLT. Our model uses erasure as a training signal inside a BIO tagger rather than as a post-hoc heuristic, and the attention-rationale baseline measures the difference.

Proposed Experiments

The primary benchmark is the Justice Connect legal problem dataset. We report exact-span F1, word F1, and two graded metrics: cross-entropy between predicted token marginals and lawyer agreement distributions, and Spearman correlation between predicted in-span probability and agreement fraction. Graded metrics matter because majority-vote F1 penalizes defensible minority highlights, the mismatch noted in iLab at Le-Wi-Di.

For zero-shot transfer, we hold out a quarter of legal categories entirely, train on the rest, and tag held-out categories from their descriptions alone. This mirrors the LITE held-out-type protocol.

For a secondary check of generality, we run SemEval-2021 Toxic Spans with the hypothesis "this comment contains insulting or abusive language." The file records only hard labels there, so this tests the entailment component, not the soft labels. The reference is the 70.83% character F1 of HITSZ-HLT, with its single BERT+CRF at 69.44%.

Three checks decide the outcome. The first is the correlation between lawyer agreement and entailment erasure effect computed before any training. If it is absent, the premise that disagreement is graded evidential strength is wrong for this data. The second is held-out-category span F1 against the attention-rationale baseline. This is the only place the entailment coupling can show value that soft labels cannot. The third is the in-category comparison of the full model, the soft-label-only ablation, and the reproduced 19.0 span-F1 baseline. It tells us whether the gain comes from treating disagreement as signal, from grounding in entailment, or from neither.

One practical risk is access. The Justice Connect data may not be publicly released. If it is unavailable, the fallback is to run the disagreement-free parts on Toxic Spans and Meta4XNLI and to state that the central claim is untested.