Structured Clinical Memory as Context for Probabilistic Decision Models: A Synthetic Evaluation with Jev

Jash Sayani, Gaurang Karwande, Achinth BharadwajarXiv 2026 · arXiv

TL;DR

This paper investigates whether providing a "structured clinical memory"—a compact, typed summary of a patient's history—improves the performance of probabilistic AI decision models. Using a synthetic benchmark, the authors found that adding this structured context increased decision accuracy from 90.0% to 93.3% and significantly improved the quality of the model's probability estimates without increasing latency.

What Problem Does This Paper Address?

Clinical decision-making relies on a patient's longitudinal history, which is typically scattered across noisy, repetitive forms and long narrative records. When using Large Language Models (LLMs) for clinical tasks, the standard approach is to pass all available raw text into the context window.

However, this forces the model to reconstruct the patient's state (e.g., "Is this condition currently active?") from scratch for every query. This is computationally expensive and error-prone. The authors propose a narrower interface: a maintained clinical memory containing structured features like condition status, onset, and certainty. They aim to determine if this pre-digested "state" actually helps a model make better, more calibrated decisions.

Methods

The researchers conducted a two-arm pilot evaluation using Jev, a model developed by TypeSafe AI that returns typed probabilistic decisions (like multiple-choice or ordinal scores) rather than free-form text.

  • Arm A (Baseline): The model received only a synthetic patient vignette.
  • Arm B (Intervention): The model received the same vignette plus a "structured clinical memory" block.

The experimental design is illustrated in Figure 1: Paired evaluation design comparing raw vignettes vs. vignettes with structured memory.

Paired evaluation design comparing raw vignettes vs. vignettes with structured memory.

The structured memory included normalized labels, longitudinal status (active, resolved, etc.), onset dates, and negation flags. The authors evaluated the model's performance using standard accuracy alongside probability metrics like Log Loss (for multiple choice) and Brier Score (for binary decisions) to measure how well the model understood its own uncertainty.

Datasets

The authors authored a new, fully synthetic benchmark for this study:

  • 30 Synthetic Cases: Spanning 12 clinical domains including cardiology, neurology, and mental health.
  • 120 Typed Questions: A mix of 56 multiple-choice, 36 binary, and 28 five-level ordinal scores.
  • Gold Standard: Labels and rationales were authored by the researchers to serve as the ground truth.

Key Insights and Ideas

  • Memory as a Reusable State: Instead of just retrieving relevant text (like standard RAG), this approach treats clinical memory as a "reusable state interface." It provides a high-fidelity interpretation of the past that the model can consume directly.
  • Better Calibration, Not Just Accuracy: The most significant gains weren't just in getting answers right, but in the model's certainty. Structured memory moved more predictions into the highest confidence bin (0.9–1.0) while maintaining high accuracy, suggesting the model becomes more "decisive" with better context.
  • Zero Latency Penalty: Despite adding an average of 310 tokens per call (a 56% increase in input size), the observed mean latency remained unchanged at approximately 0.5 seconds. This suggests that for these models, the bottleneck is not necessarily the context length within these ranges.
  • The "Oracle" Upper Bound: The authors emphasize that this is an "upper-bound" pilot because the memory was manually authored (an "oracle"). In practice, an automated extraction system would introduce errors that might degrade these benefits.

Key Results

The addition of structured memory led to improvements across all primary metrics:

  • Accuracy: Overall accuracy rose from 90.0% to 93.3%. The largest gains were seen in ordinal scores (risk levels), which improved by 7.2 percentage points.
  • Probability Quality: Choice log loss decreased from 0.258 to 0.175, and the binary Brier score improved from 0.054 to 0.022. Lower scores here indicate the model is assigning more probability to the correct options.

Detailed accuracy results by question type are shown in Figure 3: Accuracy improvements across choice, binary, and ordinal score questions.

Accuracy improvements across choice, binary, and ordinal score questions.

The improvement in probability metrics is summarized in Figure 6: Comparison of probability metrics (Log loss, Brier score, and MAE) between Arm A and Arm B.

Comparison of probability metrics (Log loss, Brier score, and MAE) between Arm A and Arm B.

Limitations and Future Work

The authors are transparent about several limitations:

  1. Synthetic Nature: The cases do not reflect the true messiness, ambiguity, or missing data found in real electronic health records (EHRs).
  2. Small Sample Size: With only 120 questions, the study is "underpowered," meaning the results are directionally positive but not yet statistically conclusive (McNemar $p = 0.219$).
  3. Oracle Memory: The study doesn't account for the errors a real-world NLP extraction pipeline would make when building the memory block.
  4. Single-Author Labels: The ground truth was not independently adjudicated by a panel of clinicians.

Future Work: The authors suggest that the next step is to test "extracted memory"—using an actual AI system to generate the memory from raw text—to see how the full error chain impacts decision quality under clinician review.

Summary updated

Abstract

Longitudinal clinical context is often distributed across noisy, repetitive forms and narrative records. We study whether a compact structured memory can improve a probabilistic decision model's answers when the underlying patient vignette is already available. We created 30 fully synthetic cases spanning twelve clinical domains, paired each with synthetically generated structured memory, and evaluated 120 typed questions with Jev. The raw-vignette arm achieved 90.0% accuracy (95% normal-approximation confidence interval: 84.6%-95.4%); adding memory achieved 93.3% (88.9%-97.8%). Five questions changed from incorrect to correct and one changed in the opposite direction (exact McNemar $p=0.219$). Choice log loss decreased from 0.258 to 0.175 and binary Brier score from 0.054 to 0.022. Memory added 310 input tokens per call on average, while observed mean latency was unchanged at approximately 0.5 seconds. These results are directionally positive rather than clinically validating: the cases are synthetic, the labels are single-author judgments, and the memory is an oracle upper bound rather than extracted production output. The pilot supports evaluating structured clinical memory as a reusable state interface for decision models.
BibTeX
@misc{sayani2026structured,
  author = {Jash Sayani and Gaurang Karwande and Achinth Bharadwaj},
  title = {{Structured Clinical Memory as Context for Probabilistic Decision Models: A Synthetic Evaluation with Jev}},
  year = {2026},
  eprint = {2611.04101},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04101},
}
Figure