Education platforms increasingly rely on simulated students to decide what a real student should do next. These include LLM agents that answer exercises "in character," knowledge-tracing networks, and memory models of forgetting. Such simulators are judged almost entirely on how well they predict the next logged answer. But a platform does not use them to predict. It uses them to choose, for example when to schedule a review or which item to serve. Those are counterfactual questions ("what would this learner recall if we waited three days instead of one?"). Logged data can answer them only under assumptions nobody can check, because the same hidden traits, such as motivation, drive both when learners practice and how well they remember.
We propose to borrow a tool from engineering digital twins and use it as a gate. A digital twin is a virtual model kept in sync with a live system so it can issue commands back to it. The tool is a falsification test, which asks whether the twin's counterfactual predictions lie outside the range that any confounding could explain. The learner twin keeps updating itself from each student's stream of answers and proposes interventions. It is allowed to act only on the decisions where it has not been statistically falsified; everywhere else it falls back to a safe default schedule. The claim is concrete and testable. Next-answer accuracy is a poor guide to whether a learner simulator's recommendations can be trusted. A falsification gate, run online, cuts harmful recommendations at little cost in learning benefit. It also gives a direct way to measure whether synthetic student data helps or hurts a twin's ability to prescribe.
Core Claim
Causal Falsification of Digital Twins shows that a simulator's interventional correctness cannot be certified from observational data without assuming no unmeasured confounding. As a practical alternative, it builds "validity envelopes" from longitudinal Manski bounds and rejects a simulator whose trajectories leave them. Applied offline to the Pulse physiology engine on MIMIC-III, it falsified the engine on 10 of 14 physiological quantities. We move this test from a one-time audit into the running loop of a learner twin, and into education, where it has not been applied.
In the simplest cross-sectional case, a binary recall outcome $Y$ under a review-delay choice $a$ has a mean potential outcome that satisfies
$$ P(Y=1, A=a) \;\le\; \mathbb{E}[Y(a)] \;\le\; P(Y=1, A=a) + P(A \ne a), $$
where $A$ is the delay the learner actually took. The bound holds whatever the confounding. A twin whose simulated recall under delay $a$ falls outside this interval, beyond sampling error, is falsified for that decision. The longitudinal version from the falsification paper extends this to histories of reviews.
Our central hypothesis has two parts. First, twins ranked highly by next-answer accuracy are frequently falsified on the specific delay or item decisions they would recommend. Second, gating actions on the envelope beats gating on predictive calibration and beats trusting the twin outright.
This addresses the actionable-insight gap as it is described for educational analytics, where pipelines produce correlations and early warnings but do not close the loop or check that an intervention helped (Exploring LMS Interaction Data). The gate's output is itself a usable signal: it tells an instructor or system designer which recommendations the model has earned the right to make, and on which concepts or delays it fails.
Research Program
The system has three parts.
The first part is the twin. Each learner's parameters are updated continuously by a particle filter, a method that keeps a cloud of weighted guesses about hidden parameters and reweights them after every observation. We adopt it because Continuous calibration of a digital twin found particle filtering about 30 times faster than static Bayesian calibration (786 s versus 22,572 s). It tracked an abrupt parameter change immediately where sliding-window calibration lagged, at 1.1% versus 0.9% RMSE. We instantiate three kinds of twin core, because the falsification question is only interesting if the candidates disagree:
- a memory model in the spirit of Enhancing human learning via spaced repetition optimization, which schedules reviews by solving an optimal-control problem over a stochastic forgetting process;
- a sequence model following Deep Knowledge Tracing;
- an LLM learner agent following Agent4Edu, which combines ability estimates, a forgetting-curve memory module, and step-by-step reasoning.
The LLM agent is too slow for per-answer updates, so it is queried asynchronously at session boundaries. Its parameters are carried between calls by the filter.
The second part is the online falsification monitor. For each decision cell, such as a delay bin crossed with a concept cluster, it maintains the envelope from accumulating logged data. It tests the twin's simulated outcomes against the envelope using a Hoeffding-based test made valid under repeated looks, via a union bound over pre-registered check times. The falsification paper uses Hoeffding tests offline; the repeated-look correction is our addition.
The third part is the actuator. It executes the twin's recommendation in unfalsified cells and a fallback policy in falsified ones. The fallback is either a threshold schedule or the memory-model optimal schedule.
The synthetic-data lens enters in a specific way. On the Equivalency, Substitutability, and Flexibility of Synthetic Data measured substitutability as how much real data synthetic data can replace on a downstream metric: 60–80% for tracking, with a 30:1 frame exchange rate. Agent4Edu showed synthetic learner traces raise adaptive-testing F1 by up to 2.39%. Both measure predictive utility. We will measure interventional substitutability: as synthetic agent traces replace a growing share of real traces in training the twin, how does the fraction of falsified decision cells change, and how does realized learning outcome change?
We expect the predictive and interventional exchange rates to diverge. Synthetic traces inherit whatever rational, homogeneous learner the generator assumes, so they may sharpen next-answer prediction while pushing counterfactuals outside the envelope. The contrastive alignment of synthetic and real sequences used by L-HAKT is the natural representative of distribution-matching fixes, and we test whether it rescues interventional validity or only predictive fit.
Baselines
The naive reference point is a do-nothing adaptive schedule: the uniform and threshold review schedules used as comparators in the spaced-repetition work. Any gated twin must at least match them on recall, or it is not worth deploying.
The strongest existing method for the scheduling decision is MEMORIZE from Enhancing human learning via spaced repetition optimization. On about 5.2 million Duolingo user-word pairs it produced statistically lower empirical forgetting than uniform and threshold schedules (P < 0.05, Mann-Whitney U). The file reports no effect size for this, so our comparison uses the same significance test on the same filtered pairs, plus effect sizes we compute ourselves.
For next-answer prediction, the strongest reported figures are:
| Model | Benchmark | Reported result |
|---|---|---|
| Deep Knowledge Tracing | ASSIST2009 | AUC 0.86 |
| L-HAKT | ASSIST2009 | AUC 80.22%, ACC 75.10% |
| Agent4Edu (GPT-3.5-turbo) | EduData | ACC 66.70%, F1 79.84% |
The two ASSIST2009 figures differ, which likely reflects preprocessing differences and is itself a warning about predictive leaderboards. We will rerun all twins under one split and one preprocessing pipeline rather than quote them. Every twin gets the same data, the same particle-filter budget, and the same hyperparameter search budget, fixed in advance and counted in GPU-hours.
Three gating baselines compete with the envelope gate. The first is trust-always: the twin acts everywhere, the natural closed-loop digital twin. The second is a calibration gate: the twin acts only where its held-out next-answer calibration error is below a threshold. The third is an off-policy-evaluation gate: the twin acts only where an inverse-propensity-weighted estimate of its policy's value beats the fallback with confidence. This is the most serious competitor, and it assumes no unmeasured confounding.
The ablation that isolates our key ingredient swaps only the gating statistic while holding the twin, filter, fallback, and data fixed. The envelope gate beats the others if, in the semi-synthetic setting where ground truth is known, it achieves a lower harm rate at equal or better mean recall. Harm rate here means the fraction of learners whose realized recall under the gated policy is below the fallback's. On real logs, beating a baseline means its falsification verdicts predict held-out interventional failure better.
Why not just…?
Why not just pick the twin with the best held-out AUC? Because AUC measures fit to the logging policy's data, and confounders such as motivation can make an observationally accurate model wrong about what happens when the schedule changes. That is the core result of Causal Falsification of Digital Twins. This alternative is a real threat, however. If AUC rank and falsification rank agree across our twins, the gate adds little. The first decisive check below measures exactly that rank agreement. We will report it even if the answer favors AUC.
Why not just run off-policy evaluation on the recommendation policy directly and skip simulators? Off-policy evaluation gives point estimates that are unbiased only if all confounders are measured and propensities are known. In Duolingo-style logs, review timing is largely user-chosen, so propensities must be estimated from the same confounded data. It also produces no model that can be updated per learner in real time. Still, it is the likeliest alternative to match us. The semi-synthetic experiment injects a hidden confounder of controlled strength so we can see where off-policy evaluation breaks and where the envelope gate is merely too conservative. If off-policy evaluation wins at realistic confounding strengths, we will say so.
Why not just measure distributional distance between synthetic and real traces, using maximum mean discrepancy? A twin can match the observed distribution of answer sequences closely and still be wrong about interventions the logs rarely contain. Distance-matching is essentially what contrastive alignment in L-HAKT optimizes. We include distance-to-real as a gating statistic in the ablation and test whether it predicts falsification.
Why not just recalibrate continuously with the particle filter and trust the tracked twin? Continuous calibration minimizes tracking error on observed data, which is still observational. The digital-twin literature notes that closed-loop synchronization rarely certifies whether counterfactual projections are causal (Closed-Loop Bidirectional Digital Twin Synchronization). Trust-always with filtering is one of our baselines, and the harm-rate comparison settles whether filtering alone suffices.
Proposed Experiments
The semi-synthetic experiment provides ground truth. We build a learner world calibrated to Duolingo review statistics, in which a hidden per-learner trait raises both practice frequency and recall. Confounding strength is swept from zero upward. Because the world's true counterfactual recall is known, we can score each gate's falsification verdicts for correctness and each gated policy for harm rate and mean recall. The loop then runs online with the particle-filtered twins. A preregistered success criterion is a lower harm rate than both the trust-always and off-policy-evaluation gates at moderate and strong confounding, with mean recall within noise of the best ungated policy.
The real-log experiments use the Duolingo spaced-repetition data, EduData from the Agent4Edu work, and ASSIST2009. On Duolingo, the decision is review delay. On EduData and ASSIST2009, the decision is which concept to practice next, and the envelope is built over concept-selection histories.
The first decisive check is the rank agreement, measured by Spearman correlation, between next-answer AUC and the number of unfalsified decision cells across all twins. We predict weak agreement; strong agreement would count against the proposal.
The second check is temporal. The gate is fit on early weeks and must predict which of a twin's recommendations are followed by worse-than-fallback outcomes in later weeks. The comparison is restricted to natural-experiment subsets where learners happened to follow or not follow the recommendation.
The third check is the interventional substitutability curve. We vary the share of Agent4Edu-style synthetic traces from 0% to 80% and plot next-answer AUC and falsified-cell fraction on the same axis. The proposal predicts a range where AUC rises while falsification also rises.
Two risks are acknowledged up front. Manski envelopes can be too wide to falsify anything when every delay bin is common, so we will report falsification power as a function of bin granularity. If power is negligible even at coarse bins, the gate reduces to trust-always, and we will report that result as a negative finding about how far assumption-free falsification can go for learning data. Second, LLM-agent latency may make the asynchronous core lag the filter; we measure recommendation staleness directly.
References
- Causal Falsification of Digital Twins
- Exploring Learning Management System Interaction Data: Combining Data-driven and Theory-driven Approaches
- Continuous calibration of a digital twin: Comparison of particle filter and Bayesian calibration approaches
- Enhancing human learning via spaced repetition optimization
- Deep Knowledge Tracing
- Agent4Edu: Generating Learner Response Data by Generative Agents for Intelligent Education Systems
- On the Equivalency, Substitutability, and Flexibility of Synthetic Data
- Towards LLM-Empowered Knowledge Tracing via LLM-Student Hierarchical Behavior Alignment in Hyperbolic Space
- Digital twin applications in urban logistics: an overview