Self-guided virtual reality (VR) therapy delivers exposure and cognitive behavioral therapy through a headset with no clinician in the room, and it works for some patients. Panic disorder symptoms fall after a 12-session avatar-guided program, and fear of heights drops with a smartphone exposure app. The open question is which patients will benefit. A natural idea is to scan the brain before treatment. Diffusion tensor imaging (DTI) is an MRI method that tracks how water moves along nerve fibers, which gives a measure of white-matter wiring. The hope is that such a scan could tell responders from non-responders.
We argue that this idea, done the usual way, will produce numbers that look good inside one clinic and fail elsewhere. Two filters cause this. First, a dataset assembled after the fact contains only patients who finished the program and finished the scans. Second, DTI measurements shift with scanner hardware. This note proposes a two-site prospective study of self-guided VR therapy for panic disorder in which every enrolled patient is tracked, including those who drop out. That full record lets us build a "retrospective-style" dataset inside it on purpose and measure how much the usual completer-only analysis inflates performance. It also lets us test whether white-matter measures add any prediction that survives transfer to a second scanner and clinic.
Core Claim
The central claim is that for automated VR therapy, the main threat to a usable response predictor is not model capacity. It is who ends up in the training data. Self-guided VR loses many patients to dropout (Shin et al.), and digital recruitment over-selects tech-comfortable, milder patients (Bentz et al.). A model trained only on completers may therefore learn to predict who stays in the program rather than who gets better. We make three testable predictions:
- Models trained on an emulated completer-only cohort will show higher internal accuracy and lower external accuracy than models trained on all enrolled patients, with dropout counted as an outcome.
- Any added value from DTI must be judged on the second site, after the analysis is locked.
- Compact tract-level summaries will transfer better than whole-brain connectivity matrices.
The novelty is the combination of three things: a known denominator, a deliberate emulation of retrospective filtering, and a locked external test, all applied to a structural brain predictor of digital therapy response. None of these has been done for VR therapy in the file's literature.
Position Relative to Prior Work
The closest work motivates the gap. Kim et al. (2025) predicted early response in panic disorder from clinical scales plus heart-rate variability and anxiety ratings recorded during VR exposure, using gradient-boosted trees. They report 85% accuracy and an AUC of 0.93 for early responders. That study had 7 early responders, internal cross-validation only, and no brain imaging, and the authors themselves name neuroimaging as the next step. Kim et al. (2022) showed that VR self-training for social anxiety changes resting-state functional networks. That is a measure of change after treatment, not a baseline predictor, and it collected no diffusion data. We take the panic-disorder setting and the VR assessment modules from the first paper and ask whether structure adds anything the cheaper signals do not.
Two papers constrain the design.
- Connectome features may not transfer. Wang et al. built white-matter connectome classifiers at one Korean hospital and tested them, locked, on the multi-site ADNI-2 cohort. For predicting conversion to Alzheimer's disease, adding connectome features to morphometry dropped accuracy from 69% to 59%, and the authors blamed differences in diffusion acquisition between sites. This is direct evidence that high-dimensional connectome features can hurt transfer. We therefore treat whole connectomes as a comparison arm, not as the default.
- Small samples overfit high-dimensional inputs. Singh et al. compress 74 tract-averaged fractional anisotropy values into a 9×9 image and train a disentangling autoencoder, matching 3D models at about 81.4% accuracy for sex classification with n=105. We reuse their low-dimensional tract representation because our cohorts will be small. They never tested it across scanners or for treatment response.
The external-validation and selection-bias literature supplies the method and the cautionary numbers:
- Jimenez-Solem et al. saw mortality AUC fall from 0.906 to 0.74 when a Danish model was moved to UK Biobank.
- Spohn et al. found only 77 of 251 prostate radiomics studies properly validated.
- Yao et al. and Zou et al. show the locked derivation-then-external-test procedure with calibration and decision curves, which we adopt.
- Cui et al. is the clearest example of the trap we emulate. Requiring a follow-up test a year later made cohort entry depend on later events.
All of these studies could only describe their filters after the fact. Our design observes the filter directly.
Research Program
We enroll consecutive adults with panic disorder at two clinics with different MRI scanners and acquisition protocols. Each patient receives the same self-guided 12-session mobile VR cognitive behavioral therapy program modeled on Panic-Relieve (Shin et al.).
Baseline measures. Before treatment, each patient has:
- clinical scales;
- a DTI scan;
- the four-module VR assessment (rest, crowded mall, relaxation, simulated hyperventilation) with heart-rate variability and in-headset anxiety ratings, following Kim et al. (2025).
Logging. Session-by-session usage is logged automatically. Patients who refuse or fail the scan are still enrolled and still treated, so the scan filter is observed rather than imposed.
Outcomes. The primary outcome uses Kim et al.'s response definition: at least a 40% drop in the Panic Disorder Severity Scale at program end. It is analyzed on everyone enrolled, with dropout counted as non-response. A secondary three-way outcome separates responders, non-responders and dropouts. Durability is checked at 6 months, which addresses the short follow-up common in this literature.
Targets. We aim for about 160 patients at the derivation site and about 100 at the external site. The external number is chosen so that a 0.05 difference in AUC is distinguishable from noise. If accrual falls short, we report confidence intervals rather than claim a difference.
DTI features. These come at three levels of compression:
- fractional anisotropy and radial diffusivity in a short list of frontolimbic tracts (uncinate fasciculus, cingulum), specified in advance as hypotheses;
- all 74 tract summaries in the grid representation of Singh et al.;
- whole-brain probabilistic tractography connectomes as in Wang et al..
Site harmonization is fitted on derivation data only, then applied unchanged.
Emulated retrospective cohort. Inside the derivation site we rebuild the dataset a chart review would produce: patients who completed the program, had a usable scan, and attended follow-up. Every model is trained twice, once on this emulated cohort and once on all enrolled patients. All models are then locked and scored on all enrolled patients at the external site. Our main summary statistic is the inflation gap:
$$G = \mathrm{AUC}_{\text{internal}}(\text{completer-trained}) - \mathrm{AUC}_{\text{external}}(\text{completer-trained}, \text{all enrolled}),$$
Here, AUC is the area under the receiver operating curve, the internal term is cross-validated within the emulated cohort, and the external term is scored on every external enrollee. We compare $G$ with the same quantity for the model trained on all enrollees.
Is DTI picking up dropout instead of response? We also fit DTI features directly to the dropout indicator. If white-matter measures predict completion, then any completer-only "DTI predicts response" finding is partly a selection effect. This is the result we most want to expose.
Baselines
Naive reference. A logistic model on baseline symptom severity alone. In practice, a clinician would guess response from how sick the patient looks at intake. Every model must beat it on the external all-enrolled cohort in AUC, calibration slope and net benefit on a decision curve.
Strongest existing method. The clinical-plus-VR-biosignal gradient-boosted tree model of Kim et al. (2025). The reported figures are 85% accuracy, F1 0.71 and AUC 0.93 for early responders, all from internal cross-validation with 7 early responders. It is the strongest competitor because it uses signals that are cheap to collect during the VR program itself, and because it already addresses the same disorder and the same response definition. We reimplement it on our data with the same features, the same tuning budget (an identical nested cross-validation grid search) and the same locked external test as every DTI model. We do not compare against the 0.93 directly, because that figure comes from a different, tiny internal sample. Beating this baseline means a DTI-augmented model whose external AUC lower confidence bound exceeds the biosignal model's external AUC, with no worse calibration.
Ablations. Three ablations isolate the proposal's ingredients:
- Selection ablation: completer-trained versus all-enrollee-trained versions of each model.
- Representation ablation: frontolimbic tracts versus 74-tract grid versus full connectome. This tests the transfer failure reported by Wang et al..
- Harmonization ablation: with versus without site harmonization.
Why not just…?
Why not just skip the scanner and use clinical plus in-VR biosignals? This is the real threat. Heart-rate variability and anxiety ratings come free with the therapy, and Kim et al. (2025) already reach high internal accuracy with them. The study is built so that this question can be answered either way. If DTI adds less than about 0.03 external AUC over the biosignal model, we will report that the scan is not worth its cost for this decision. A credible negative result here is as useful as a positive one. The selection-bias findings stand regardless of whether DTI earns its keep.
Why not just reweight completers by their estimated chance of finishing? Inverse-probability weighting corrects only for dropout predictors that were recorded. As Zou et al. note for registry data, it is blind to unrecorded ones, such as technology reluctance (Bentz et al.). Because we observe the full denominator, we can run a weighted completer model as an extra arm and check directly how much of the inflation gap it closes. In most retrospective settings this check is impossible.
Why not just pool both sites and cross-validate on a larger sample? Pooling mixes the scanner and clinic difference into the training data, so cross-validation measures reproducibility rather than transfer. The 0.906-to-0.74 drop in Jimenez-Solem et al. and the prognostic drop in Wang et al. are what pooled estimates hide. We report pooled cross-validation alongside the locked external test to show the size of that difference.
Why not just use functional MRI as in the social anxiety study? Kim et al. (2022) measured network change after treatment, which cannot be used to choose patients beforehand. Whether baseline structure transfers across sites better than baseline function is our hypothesis, not an established fact. We therefore keep functional imaging out of scope rather than claim superiority.
Decisive Checks
The selection claim is supported if the completer-trained models show an inflation gap $G$ clearly larger than the all-enrollee models. It is also supported if DTI features predict dropout above chance. The DTI claim is supported only if a tract-level model beats the biosignal baseline on the locked external cohort, keeps calibration slope near one, and adds net benefit on decision curves at the response probabilities clinicians would act on. The representation claim from Wang et al. is supported if the full connectome transfers worse than the compact tract summaries. Each outcome is pre-registered with the locked pipeline before the external data are unblinded. Any of these results, including the null, tells a clinic whether to put self-guided VR patients through a scanner.
References
- Effectiveness of Self-Guided Virtual Reality–Based Cognitive Behavioral Therapy for Panic Disorder: Randomized Controlled Trial
- Effectiveness of a stand-alone, smartphone-based virtual reality exposure app to reduce fear of heights in real-life: a randomized trial
- Simulated virtual reality experiences for predicting early treatment response in panic disorder
- Alteration of resting-state functional connectivity network properties in patients with social anxiety disorder after virtual reality-based self-training
- Diagnosis and prognosis of Alzheimer's disease using brain morphometry and white matter connectomes
- An Interpretable Representation Learning Approach for Diffusion Tensor Imaging
- Developing and validating COVID-19 adverse outcome risk prediction models from a bi-national European cohort of 5594 patients
- Radiomics in prostate cancer imaging for a personalized treatment approach - current aspects of methodology and a systematic review on validated studies
- A novel radiomics based on multi-parametric magnetic resonance imaging for predicting Ki-67 expression in rectal cancer: a multicenter study
- Predictive value of current nodal staging systems and development of machine learning nomogram for resectable pancreatic head cancer: a population-based study and multicenter validation
- A novel computed tomography radiomic nomogram for early evaluation of small airway dysfunction development