Calibrated Coincidence Detection for Audiovisual Awareness Tests

Some patients with disorders of consciousness cannot move or speak. Clinicians can still test whether such a patient follows commands by flashing and speaking numbers, asking the patient to attend to one of them, and checking whether the brain's electrical response (recorded on the scalp as EEG) singles out the attended number. The current practice trains a standard classifier on these responses and declares awareness if its accuracy beats chance on a binomial test. That decision has two weak points. The classifier reads a fixed time window after each stimulus, which breaks when auditory responses arrive late. The binomial test assumes trials are independent, which fails when a patient's arousal drifts across a session. A false "aware" verdict, or a missed one, matters clinically. Rare-event physics has dealt with the same structure for decades: a faint signal seen in two channels that are not perfectly synchronized, buried in a non-stationary background, with an expensive cost for a false discovery.

We propose to borrow two specific tools from that field. The first is a coincidence likelihood ratio that treats the visual and auditory parts of the brain response as two "messengers," each with its own uncertain arrival time. The second is time-slide background estimation, which measures how often the detector fires on data that cannot contain a real response. In plain terms, we replace "classifier accuracy beats a binomial threshold" with "a latency-tolerant evidence score whose false-alarm rate was measured on the subject's own scrambled data." The bet is that this detects command following in fewer trials at a false-alarm rate that is actually verified.

Core Claim

For gaze-independent audiovisual paradigms, the decision statistic should be a likelihood ratio that integrates over plausible per-trial latencies of the visual and auditory response components. Its null distribution should come from the session's own data, re-aligned to wrong stimulus times. We predict this statistic will reach a fixed detection sensitivity with fewer stimulus repetitions than a fixed-window SVM or LDA, while holding the empirical false-positive rate at its nominal level on sham sessions. Sham sessions are those in which healthy subjects are told not to attend. The binomial-on-accuracy procedure is not expected to hold that rate under drifting arousal.

Why This Intersection, and What Prior Work Leaves Open

The audiovisual evidence motivates the latency model. A Novel Audiovisual Brain-Computer Interface and Its Application in Awareness Detection presented flashing numbers together with matching spoken numbers. It reports online SVM accuracy of 95.67% audiovisual versus 86.33% visual-only and 62.33% auditory-only in healthy subjects, and 66–74% in 5 of 7 patients, judged by a binomial test. A gaze-independent audiovisual brain-computer Interface for detecting awareness of patients with disorders of consciousness carried this design to the bedside. Neither paper changes the fixed-window decoder or the significance test. That is the part we target.

Comparison of P300 Responses in Auditory, Visual and Audiovisual Spatial Speller BCI Paradigms supplies the key negative result. Auditory responses there had clear scalp topographies, yet a 0–600 ms LDA reached only 52.5% (SD 27.2%) because their latencies shifted. Influence of Auditory Cues on the Neuronal Response to Naturalistic Visual Stimuli in a Virtual Reality Setting adds a constraint: adding sound suppressed the early visual P1 peak (about 1.3×) and enhanced and delayed later peaks (N2c about 1.25×, P3b about 1.5×). An audiovisual response therefore cannot be modeled as the sum of two unimodal templates. Our model fits audiovisual templates and latency priors directly rather than assuming they add.

The physics side supplies the method. Enhancing the significance of astrophysical events with multimessenger coincidences builds an exact likelihood ratio from a temporal-offset prior, spatial overlap, and channel-specific background rates. With it, a weak gravitational-wave trigger's posterior rose from 7.7% to 86% given a coincident neutrino. That paper also warns that the gain is very sensitive to the signal priors, which is why we estimate latency priors from data rather than set them by hand. Gravitational-wave practice of measuring false-alarm rates on deliberately misaligned data motivates our time-slide nulls, and Study of background from accidental coincidence signalsin the PandaX-II experiment* shows the same concern in detectors: chance coincidences set a floor that must be measured, not assumed away.

The closest existing bridge is Cross-modal Background Suppression for Audio-Visual Event Localization. It gates audio and video features against each other, treating any segment where the two streams disagree as background, and reaches 79.7% on the AVE video benchmark. It uses synchronous frame-level gates on high-SNR video and spectrograms, and it offers no calibrated false-alarm control. Both choices fail for EEG, where cortical latencies differ between modalities and the signal sits below the noise. Noise Suppression and Surplus Synchrony by Coincidence Detection gives the theoretical intuition that coincidence gates suppress independent noise, but only at the spiking-circuit level.

We deliberately do not import pulse-shape or particle-track analogies such as prompt-light fractions. The ERP evidence identifies latency, not pulse kinetics, as the failure mode.

Research Program

Each stimulus presentation $i$ yields a spatially filtered EEG epoch $x_i(t)$. Under the target hypothesis, the epoch contains an audiovisual response template $\phi$ shifted by an unknown latency $\tau$ drawn from a prior $p(\tau)$, plus background noise with covariance $\Sigma$ estimated from non-target epochs. The per-trial evidence is

$$\Lambda_i = \frac{\int p(\tau)\, \mathcal{N}\big(x_i;\, a\,\phi(t-\tau),\, \Sigma\big)\, d\tau}{\mathcal{N}(x_i;\, 0,\, \Sigma)},$$

where $a$ is a fitted amplitude. We use separate early (visual-dominated) and late (multisensory) template components, each with its own latency prior, so that the late component can shift independently of the early one. Each component's evidence combines in a product form, which is the multimessenger coincidence term. A small number of mixture weights also allows only one component to be present, because some patients may retain auditory but not visual processing.

The session statistic is $S = \sum_i \log \Lambda_i$, summed over target presentations and contrasted with non-target ones. Templates and latency priors are fit hierarchically. A group-level prior learned from healthy subjects is shrunk toward each new subject using a few minutes of data, which addresses the calibration bottleneck without deep networks.

The null distribution of $S$ comes from time slides. We recompute the statistic after shifting the stimulus-label sequence by offsets longer than one response. This keeps the session's real autocorrelation, artifacts, and arousal drift, while destroying any genuine stimulus-locked response. The awareness verdict is a time-slide p-value, which we compare against both the binomial test and a label-shuffle permutation test.

Baselines

The naive reference is the protocol of A Novel Audiovisual Brain-Computer Interface and Its Application in Awareness Detection: a fixed-window linear SVM with a binomial test on accuracy. That paper reports 95.67% online accuracy (healthy, audiovisual) and 66–74% in responsive patients. We rerun it on identical epochs and trial counts. Beating it means equal or higher detection sensitivity at fewer repetitions, with a false-positive rate on sham sessions at or below the nominal 5%.

The strongest classical competitor is xDAWN-style spatial filtering with shrinkage LDA, the standard P300 pipeline. The file has no reported number for it on these paradigms. On spatial audio, the related LDA result is 91.3% audiovisual and 52.5% auditory in Comparison of P300 Responses in Auditory, Visual and Audiovisual Spatial Speller BCI Paradigms. It receives the same spatial filter, the same calibration data, and a nested cross-validated search over window length and shrinkage, with a matched number of tuning configurations.

The strongest learned competitor is EEGNet. BCIAUT-P300: A Multi-Session and Multi-Subject Benchmark Dataset on Autism for P300-Based Brain-Computer-Interfaces reports 92.3% for it with 1,386 parameters. It is trained with the same subject data, optionally pretrained on the same healthy pool that supplies our group prior, and given equal tuning effort. We also include sensor-weighted SVM from sw-SVM: sensor weighting support vector machines for EEG-based brain–computer interfaces, which reported 80.71% versus 70.71% for plain SVM in a sparse-trial setting, as a low-data spatial-weighting competitor.

Three ablations isolate the ingredients. The first fixes $\tau = 0$ (no latency marginalization). The second uses one additive template instead of separate early and late components. The third swaps the time-slide null for a label-shuffle permutation null. Woody-style per-trial latency alignment followed by LDA, a decades-old EEG technique, serves as a direct test of whether marginalizing over latency beats simply estimating it.

Why not just…?

Why not just widen the classifier window or align latencies first? This is the most serious threat to the latency half of the proposal. If Woody alignment plus shrinkage LDA matches our auditory-condition accuracy at low repetition counts, the latency-marginalization claim fails and we will say so. We expect marginalization to help most at one to three repetitions. At that point a single trial is too noisy to estimate its own latency reliably, whereas integrating over the prior does not require a point estimate. The Chang et al. auditory condition is where this comparison is decided.

Why not just train EEGNet end to end? It may well match us on healthy subjects with ample data. The proposal does not rely on beating it there. The relevant regime is minutes of data from a fatiguing patient, where the BCIAUT authors flag training cost as an obstacle. More importantly, a network score has no calibrated null of its own. We will in fact apply time slides to EEGNet's output too. If time-slided EEGNet matches our false-positive control and sensitivity, the contribution reduces to the calibration procedure, which is still useful but smaller.

Why not just use a permutation test instead of the binomial? Label-shuffle permutation tests already exist. They break the temporal structure of arousal drift, so their nulls may be too narrow when trials are correlated. Our sham-session experiment measures this directly. If shuffle and time-slide nulls give the same false-positive rates, we drop the time-slide claim and keep the simpler test.

Why not rely on language priors, as in A P300 BCI for the Masses: Prior Information Enables Instant Unsupervised Spelling? That work raised 3-repetition spelling accuracy from 24.6% to 74.8% using trigrams. Awareness tests have no text to exploit, so the approach does not apply to the main target. In the speller experiment, its prior would multiply into ours rather than compete with it.

Proposed Experiments

The first experiment tests latency on spatial audio. We will use the Chang et al. hiragana spatial speller data if it is obtainable, or a matched re-collection with 16 subjects and five azimuths otherwise. We report accuracy and information transfer rate against repetitions for visual, audiovisual, and auditory conditions. The decisive check is whether auditory accuracy at three or fewer repetitions improves substantially over the 52.5% LDA reference and over the Woody-aligned baseline.

The second experiment is new data collection and carries the clinical claim. Healthy subjects run the Wang et al. congruent number paradigm in two conditions: attend (true command following) and sham (passive listening or a distractor task), with arousal manipulated through long sessions. The empirical false-positive rate of each verdict procedure on sham sessions is the primary outcome. The second outcome is trials-to-detection at a matched 5% false-positive rate on attend sessions. A clear success means our statistic holds 5% on sham while binomial-on-accuracy exceeds it, and it detects attenders in fewer trials than every baseline. If patient data under the Wang et al. protocol becomes available, we report how verdicts agree and disagree across procedures. Without ground truth we will not claim improved diagnosis there.

The third experiment checks generality on the VR hazard paradigm of Al Boustani et al., where the file reports no SVM accuracy number. It tests whether separate early and late templates recover the subtle "burning" events better than an additive template. That result would directly confirm or refute the non-additivity premise behind the model.