Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understanding

Fan Zhang, Yankai Chen, Zhuohan Xie, Yixi Zhou, Sijia Peng, Lei Fan, Xinhua Ji, Cunyuan Zheng, Huangyong Shan, Philip S. Yu, Xue Liu, Yu Chen, Preslav Nakov, Songwei HearXiv 2026 · arXiv

TL;DR

Standard NLP evaluation metrics can lie by hiding individual decision flips behind stable aggregate accuracy scores. This paper evaluates Jev (a decision-native model) alongside nine generative LLMs on legal contract review, revealing that minor prompt variations cause massive instability in individual predictions. To address this, the authors introduce a multi-dimensional framework evaluating cost, latency, and "persistent correctness."

What Problem Does This Paper Address?

Contract review is a classic "multi-decision workload": a single long document requires a model to make multiple independent judgments (e.g., "Is this information confidential?", "Are copies permitted?"). Typically, AI models are evaluated using aggregate accuracy across a dataset.

However, the authors identify a critical flaw in this approach: aggregate accuracy conceals underlying decision instability. A model can achieve an identical 75% accuracy score under two different prompt configurations, yet have flipped 10% of its individual decisions because its new errors (regressions) and new correct answers (corrections) perfectly canceled each other out. Furthermore, high agreement across repeated runs doesn't guarantee reliability; a model can be consistently wrong. This paper tackles the challenge of identifying how stable individual decisions remain when prompt configurations—such as hypothesis visibility, requested outputs, and output order—change.

Methods

The authors compare Jev (v1.13.0), an architecture designed specifically for bounded-choice classification without free-form generation, against nine generative language models (including variants of Qwen3.5, Gemini, GPT, and Claude).

To evaluate decision stability, they design a controlled experiment centered on a fixed "anchor" judgment across four distinct request conditions:

  • Condition A: Only the target anchor hypothesis is visible and requested.
  • Condition B: All 17 contract hypotheses are visible, but only the anchor label is requested.
  • Condition C: All 17 hypotheses are visible and requested, with the anchor placed first.
  • Condition D: All 17 hypotheses are visible and requested, with the anchor placed last.

Each condition is repeated three times (resulting in 12 total responses per anchor) to measure persistent correctness (whether a target is answered correctly across all 12 variations) and to track label-flipping behavior. The authors also evaluate real-world operational trade-offs by measuring financial cost and client-side response time (latency).

Figure 1: Overview of the evaluated workload and the questions studied. (1) ContractNLI poses many structured judgments about one shared document; hypotheses are paraphrased for illustration. (2) A fixed development-set judgment (contract 595, hypothesis nda-2) changes from consistently correct to consistently wrong when Qwen3.5-4B, with thinking disabled, sees additional non-target hypotheses; the contract, requested output, and inference settings remain unchanged. (3) On the test stability panel, Qwen3.5-4B has identical Condition A and B accuracy, yet nine of 90 paired judgments change. (4) Jev occupies the lowest-cost point among the compared models while hosted models reach higher baseline accuracy (Section 4). (5) We therefore evaluate cost, response time, average correctness, and persistent correctness together.

Datasets

The evaluation is conducted on ContractNLI, a benchmark dataset for document-level natural language inference on contracts.

  • Baseline Accuracy Evaluation: Uses the official test split containing 123 contracts and 2,091 labeled judgments.
  • Stability Panel: Uses a preselected subset of 30 contracts (with one anchor hypothesis per contract) to run the intensive 12-replicate stability mapping.

Key Insights and Ideas

  • The Illusion of Aggregate Accuracy: Identical macro-scores hide chaotic decision-level flipping. Opposing corrections and regressions frequently cancel each other out in aggregate metrics.
  • Context Pollution: Simply making non-target hypotheses visible in the prompt context (Condition B vs. Condition A) can cause a model's prediction on the exact same target question to flip from consistently correct to consistently wrong.
  • Consistency $\neq$ Correctness: High agreement across runs often just signifies a "stable error." Treating unvarying repetition as a proxy for stability can mistakenly validate a model that is persistently incorrect.
  • Architecture Dictates Efficiency: Because Jev bypasses autoregressive free-form text generation in favor of native choice selection, it operates on a completely different cost-accuracy frontier than generative LLMs.

Key Results

  • Cost & Speed: Jev is the clear efficiency leader, achieving the lowest inference cost ($0.000228 per contract) and the fastest median baseline response time (1.24 seconds).
  • Baseline Accuracy: Hosted LLMs achieved higher overall baseline accuracy, led by Gemini 3.1 Pro Preview at 83.21% (compared to Jev's 77.38%).
  • Persistent Correctness: Claude Sonnet 5 demonstrated the highest decision stability, keeping 24 out of 30 anchors correct across all 12 request variations. Jev was highly competitive, retaining 23 out of 30 anchors.
  • Ranking Discrepancies: A model's ranking by baseline accuracy did not correlate with its ranking by persistent correctness, proving that standard benchmarks fail to capture decision stability.
Figure 2: Jev 1.13.0 occupies the lowest-cost end of this comparison, while Gemini 3.1 Pro Preview attains the highest baseline accuracy. Coordinates summarize 123 official-test contracts (2,091 judg- ments); bubble area gives the fraction of 30 preselected anchors in distinct test contracts answered correctly in all four conditions and three repeats. All 360 planned stability requests per model enter the evaluation, and invalid responses cannot satisfy All 12 correct. The dotted frontier connects nondominated point estimates under the cost accounting in Section 3. Tables 1 and 3 report confi- dence intervals; the connecting segments do not represent intermediate configurations or significant differences.

Limitations and Future Work

The stability analysis relies on a relatively small panel of 30 contract targets drawn from a single domain (ContractNLI), meaning the findings may not immediately generalize to other legal tasks or broader document types. Additionally, because the API models were evaluated via commercial gateways, the recorded latencies and costs reflect specific provider infrastructure and pricing models that are subject to change.

The authors suggest that future work should include human performance baselines for comparison and extend this multi-decision stability framework to other complex, long-context document understanding tasks.

Summary updated

Abstract

Contract inference requires multiple judgments about a shared document, but aggregate accuracy can conceal changes in the individual decisions. Repeated agreement is also insufficient: a model may consistently return the wrong answer. In this paper, we compare Jev with nine language models on ContractNLI, evaluating inference cost, response time, average correctness, and correctness across repeated request conditions. Controlled comparisons vary hypothesis visibility, requested outputs, and output order while keeping the contract and target judgment fixed. Jev has the lowest cost and median response time among the evaluated configurations, while hosted language models achieve higher baseline accuracy. Rankings by baseline accuracy differ from rankings by correctness across every condition and repeat, although small differences in the latter do not establish a general stability advantage. Development diagnostics further reveal compensating corrections and regressions, as well as persistent errors. These findings motivate evaluating cost and response time alongside whether individual judgments remain correct as the request configuration changes. Code: https://github.com/ZF-Utokyo/Jev-Benchmark
BibTeX
@misc{zhang2026same,
  author = {Fan Zhang and Yankai Chen and Zhuohan Xie and Yixi Zhou and Sijia Peng and Lei Fan and Xinhua Ji and Cunyuan Zheng and Huangyong Shan and Philip S. Yu and Xue Liu and Yu Chen and Preslav Nakov and Songwei He},
  title = {{Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understanding}},
  year = {2026},
  eprint = {2609.27678},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.27678},
}
Figure