Observational Withdrawal: Removing Tutoring Support in Stages to Build Skills That Last

AI tutors and assistants tend to make people better at a task while the help is switched on, and sometimes worse once it is switched off. The standard response is to fade the help gradually. Fading is usually driven by how well the learner performs with help, and the evidence is only checked minutes or an hour later. This proposal borrows a practice from tunnel engineering. There, support is removed or excavation advances one stage at a time. After each stage, engineers watch the ground's short-term movement settle before taking the next step. The equivalent for learning is to remove assistance in discrete steps, measure how far unassisted performance drops right after each step and how quickly it recovers, and advance only once that recovery has levelled off.

The claim is twofold. First, the size of the drop and the shape of the recovery after each withdrawal step predict retention weeks later better than assisted performance does. Second, a controller that gates each step on observed stabilization will produce better 28-day unassisted skill than continuous adaptive fading, cost-throttled help, or fixed schedules. Both claims are testable in a multi-week human study, and the first can also be probed offline on existing tutoring logs.

What each direction contributes, and what is borrowed honestly

The geotechnical literature supplies a way of working, not a physical law of learning. Three results matter here. The first is that staged construction produces a path-dependent final state. A process-load method for extra-large tunnels computes loads stage by stage using coefficients for how each pilot tunnel affects its neighbours. It predicted 107.35–115.83 kPa against a measured 94.34 kPa, while final-state design gave 125.81–151.24 kPa. The second is that deformation is concentrated in specific stages. In a Pile-Beam-Arch subway station model, pilot-tunnel excavation caused about 67% of surface settlement and arch installation 23.1%. The third is that each stage produces a recurring three-phase response: rapid movement, decelerating growth, then stabilization, reported in Wu et al., Chen et al., and Zhang et al.. Back-analysis from structural deformation adds the idea of inferring hidden loads from observed displacement, with errors under 10% across 56 tunnels.

We reuse three things from this work. We withdraw support in stages, we watch each stage's transient before advancing, and we infer hidden state from observed movement. We do not claim that minds obey a mechanical constitutive law. The file's more literal mappings, such as stress redistribution across a "cognitive network" or damage variables, have no evidence behind them, and we set them aside.

The learning literature motivates the proposal. Wu et al.'s logic-puzzle study found three things:

  • Assisted performance overestimated later unassisted ability by 0.22 reward-rate units.
  • People who used AI scored 3.42 in the unassisted phase versus 3.86 for those who did not.
  • Time spent reasoning alone before asking for help ("solo share") predicted learning, whereas request counts did not.

In short, the signal that matters is what learners do without support, which is exactly what a withdrawal step exposes.

Skill-Adaptive Ghost Instructors supplies the strongest fading design. Its guidance fades quickly on success and returns slowly on error, driven by a smoothed error signal with fixed weights. It beat static guidance at a 10-minute retention test. However, its fading is continuous and driven by assisted performance, and it never measures beyond 10 minutes. Proximal State Nudging tapers shared control toward states where the gap between an expert's and the human's expected return is large. It needs a near-optimal expert policy and was evaluated within an hour. PSI-KT models knowledge as a mean-reverting process with interpretable forgetting and consolidation traits. It only predicts and does not distinguish correct answers produced with help from those produced without. Our measurement model builds on it.

Core Claim

Support is held at discrete levels $s_k$, where $s_0$ is full help and $s_K$ is none. When the controller moves from level $k-1$ to $k$, we record the learner's unassisted-probe or at-level performance $y_k(t)$ over subsequent trials $t$. We fit a three-phase response:

$$y_k(t) = y_{k}^{\infty} - \Delta_k\, e^{-t/\tau_k}$$

Here $y_k^{\infty}$ is the level the learner stabilizes at, $\Delta_k$ is the size of the immediate drop, and $\tau_k$ is the recovery time constant. The exponential form is the simplest curve with a fast initial change that decelerates to an asymptote, matching the reported three-phase shape. We treat it as a working model and will check its fit, not assume it. The controller advances to $k+1$ only when the predicted remaining recovery $\Delta_k e^{-t/\tau_k}$ falls below a tolerance $\epsilon$. If the drop exceeds a safety bound, it steps back one level, not to full support.

The measurement claim is about what we call withdrawal debt, $\sum_k (\bar y_{k-1} - y_k^{\infty})$. Here $\bar y_{k-1}$ is performance just before step $k$, so the debt is the total performance that did not recover after each step. We expect this sum to predict 28-day unassisted ability beyond what assisted accuracy and solo share predict. The intervention claim is that gating steps on stabilization beats both continuous fading and ungated stepping at 28 days.

Research Program

The work has three parts.

The first is an offline analysis. Using tutoring logs in which help use is recorded per attempt, we test whether the first unassisted attempts after a run of assisted attempts carry more predictive signal for later correctness than the assisted attempts do. We will verify which of Assist12, Assist17, Junyi15, and XES3G5M actually record hint or assistance flags. The file does not say, and datasets without flags will be dropped from this analysis. We extend PSI-KT with a per-attempt assistance level entering the observation model, and add the "post-withdrawal drop" as a feature.

The second is building the controller. It is a small stateful policy that works with any tutor exposing discrete help levels, with $\epsilon$ and the safety bound tuned on pilot data.

The third is a longitudinal human study. It measures retention at 10 minutes, 1 day, 7 days, and 28 days. This fills the gap every cited scaffolding study leaves open.

Baselines

The naive reference is unassisted self-practice: the same task and time budget with no help. It shows whether any support is worth its cost. PSN reports that self-practice produced 50% more collisions in parking training than their method, so we also record errors during training, not only retention.

Static assistance means help is always available at a fixed level. It is the setting in which Wu et al. observed deskilling (3.42 vs. 3.86 reward rate). Our method should beat it clearly at every delay. Failing to do so would falsify the premise.

The strongest competitor is continuous asymmetric fading, reimplemented from Ghost Instructors for our task. Its help level is a smoothed function of recent error that drops fast on success and recovers slowly on error. It is the closest design and the best reported retention result: about 99% vs. about 88% pitch accuracy and about 96% vs. about 77% fingering accuracy against static guidance at a 10-minute test (N=30). For a fair comparison, both controllers get the same help levels, the same session length, and the same number of pilot participants for tuning their parameters (the smoothing rates for this baseline; $\epsilon$ and the step size for ours). Beating it means a statistically reliable advantage in latent unassisted ability at 28 days, estimated with Wu et al.'s Bayesian latent-ability model. Parity at 10 minutes would be acceptable.

Cost-throttled on-demand help follows Wu et al., where charging for each request cut requests from 6.67 to 3.33. It is the cheapest behavioural alternative.

Two ablations isolate the key ingredient, the stabilization gate. The fixed-schedule stepped arm uses the same discrete levels but advances every $n$ trials, with $n$ matched to our arm's median. The probe-only arm inserts identical unassisted probes and uses them to decide help but always returns the learner to full support after a probe.

On the offline side, the baselines are AKT, IEKT, and PSI-KT. XES3G5M reports IEKT as best among 19 models. No reported absolute AUC for IEKT was found on Lacuna, so we will rerun all models ourselves under one split and tuning budget. For the motor-domain replication, the baseline is PSN on CARLA parallel parking. It reports 7x larger unassisted gains than blended shared autonomy, and no absolute value was found on Lacuna.

Why not just…?

Why not just use continuous adaptive fading, which already works? This is the real threat. Continuous fading may track the same information our gate uses. Its reported advantage exists only at 10 minutes, and it reacts to performance while help is still partly on, which is the signal Wu et al. show inflates estimates of ability. If the two arms tie at 28 days, we will report that staging adds nothing beyond good fading, and the contribution reduces to the measurement result and the longitudinal data.

Why not just add a hint flag to a knowledge-tracing model? That is part of what we do offline, and it may capture most of the predictive value. The withdrawal-debt hypothesis is specifically that the transient after removal adds signal beyond per-attempt flags. The comparison between PSI-KT with an assistance flag and the same model plus post-withdrawal drop features settles this directly. A logging model also cannot choose when to withdraw help, and the human study tests that choice.

Why not just insert periodic unassisted tests? Retrieval probes are cheap and established. This alternative is exactly our probe-only ablation. The difference we test is commitment: in our arm the learner stays at the reduced level, bearing more of the load, instead of snapping back to full help. If probe-only matches our arm, commitment is not what matters, and we will say so.

Why not just remove help entirely? Self-practice is included as an arm. The PSN results suggest it costs safety and frustration during training, but it could win on long-term retention, and our 28-day measurement would show that.

Proposed Experiments

The primary study uses Wu et al.'s logic-puzzle paradigm with graded help levels, from full solution down to a single-constraint hint and then none. It is run online over four weeks with seven arms: self-practice, static, cost-throttled, continuous fading, fixed-schedule stepped, probe-only, and gated stepped. Each arm has roughly 40 participants, which is of the order Wu et al. used across conditions, and the number will be refined from pilot effect sizes. The primary outcome is latent unassisted ability at 28 days. Secondary outcomes are the 1-day and 7-day curves, errors during training, time on task, and self-reported workload.

Four checks decide the proposal. First, does the three-phase curve fit individual post-withdrawal trajectories better than a single step change or linear recovery (judged by held-out likelihood)? If not, the gating rule has no basis. Second, does withdrawal debt predict 28-day ability after controlling for final assisted accuracy and solo share? Third, does the gated arm beat continuous fading and both ablations at 28 days? Fourth, does the offline drop feature improve held-out prediction of delayed correctness in the logs? A secondary replication on CARLA parking against PSN tests whether gating works without an expert policy. The same checks then determine whether it transfers to motor skill.