Beyond Accuracy: Information-Theoretic Decoder Evaluation Under Latent Intent

Tasha Kim, Benjamin Ballyk, Anson Lei, Ingmar Posner, Oiwi Parker JonesarXiv 2026 · arXiv

TL;DR

Standard accuracy metrics fail to predict how well a brain-controlled robot will actually perform because they ignore the sequential nature of control. This paper introduces an information-theoretic metric, the Cumulative Identification Envelope ($U_{id}$), which identifies a sharp "phase transition" in task performance: if a decoder can't distinguish even one pair of goals, robot performance degrades linearly over time, regardless of how high the overall accuracy is.

What Problem Does This Paper Address?

In assistive robotics—like a wheelchair controlled by brain signals (EEG)—a decoder translates noisy neural activity into a "goal" for the robot. Traditionally, researchers evaluate these decoders using standard classification accuracy.

However, assistive control is sequential. A robot planner doesn't just act on one signal; it accumulates evidence over many time steps. The authors argue that accuracy is a poor proxy for task performance (measured as "regret," or the extra cost incurred compared to an oracle that knows the user's intent). A decoder could be 90% accurate but consistently confuse "Turn Left" with "Turn Right," leading to a total failure in navigation. Conversely, a 50% accurate decoder that makes diverse, non-systematic errors might allow a planner to eventually reach the correct goal with very little extra effort.

Methods

The authors formalize brain-to-robot control as a Partially Observable Markov Decision Process (POMDP). Their key innovation is shifting focus from aggregate accuracy to the "hardest" part of the problem: distinguishing the two most similar goals.

  1. Pairwise Chernoff Information ($\Gamma_{ij}$): They use this information-theoretic measure to quantify how difficult it is for a decoder to distinguish goal $i$ from goal $j$ based on its confusion matrix.
  2. The Bottleneck ($\Gamma_{min}$): They identify the minimum Chernoff information across all pairs of goals as the critical factor governing performance.
  3. Cumulative Identification Envelope ($U_{id}$): They derive a closed-form predictor that estimates the total expected identification error over time. Crucially, this can be computed directly from a standard confusion matrix in $O(K^2)$ time, where $K$ is the number of goals.

Datasets

The researchers validated their theory across diverse domains to ensure the results weren't specific to one type of sensor:

  • BCI (Brain-Computer Interface): Two major EEG motor imagery datasets, Stieger2021 (62 subjects) and BNCI2014001 (9 subjects).
  • Multimodal Sensing: The NinaPro DB2 dataset, which includes EMG (muscle signals), accelerometers, and data gloves.
  • Computer Vision Proxy: CIFAR-100, used to simulate decoders spanning "orthogonal" information channels (like color vs. texture).

Key Insights and Ideas

  • The Phase Transition: The paper proves a "sharp phase transition" in regret. If all rows of a confusion matrix are distinct ($\Gamma_{min} > 0$), the extra cost (regret) stays bounded even as the task goes on forever. If any two rows are identical ($\Gamma_{min} = 0$), regret grows linearly with time.
  • The "Accuracy Trap": Accuracy is an aggregate metric that can hide fatal flaws. As shown in Figure 1, two decoders can have the exact same validation accuracy but result in vastly different real-world performance.
Figure 1: Decoder distinguishability governs planning regret. (Left): Five confusion-matrix channels with identical decoder accuracy (α=0.50, K=4). Varying Γmin produces 7.2× empirical cumulative regret (22.7→162.6) at T=600. Dashed lines: theoretical envelope ∆ubUid(C). (Right): Real EEG, illustrative (BNCI2014001 Subject 5): two decoders with identical validation accuracy (20/57 trials), but Uid=99.4 vs. 170.5. Held-out replay (accuracy 48.3% vs. 34.7%) gives 11.3 vs. 108.8 regret at T=600, as Uid predicts.
  • Orthogonality Matters: $U_{id}$ is most powerful when comparing decoders that use different types of features. When decoders share the same underlying signal features, accuracy and $U_{id}$ tend to agree. But when features are independent (e.g., combining eye-gaze with EEG), $U_{id}$ is a much better predictor of which sensor suite will actually help the user reach their goal.

Key Results

  • Predictive Power: On the CIFAR-100 benchmark, $\log(U_{id})$ explained 89% of the variance in cumulative regret, whereas accuracy alone was a much weaker predictor.
  • Real-World EEG Failure: In a striking example from the BNCI2014001 dataset (Subject 5), two decoders achieved an identical 35.1% validation accuracy. However, when used for control, one decoder incurred 9.6x more regret than the other. $U_{id}$ successfully predicted this massive performance gap where accuracy was blind.
  • Bound Verification: The authors showed that their theoretical $U_{id}$ envelope acts as a reliable upper bound for regret on real-world data, holding for 100% of cases in the BNCI dataset and ~86% in the larger Stieger dataset.
Figure 3: Empirical verification showing that the theoretical $U_{id}$ envelope (dashed line) successfully bounds the empirical regret across hundreds of BCI test conditions.

Limitations and Future Work

  • Action-Independence: The current model assumes the user's signals don't change based on what the robot is doing (no "active sensing"). In reality, users often adapt their strategy if they see the robot failing.
  • Temporal Correlation: The proof assumes observations are independent over time (i.i.d.). Real neural signals often have temporal dependencies (autocorrelation) that might make the bounds more conservative.
  • Deployment: While tested in a high-fidelity "reaching" simulation, the authors note that $U_{id}$ should be further tested in complex, deployed stacks like wheelchair navigation or robotic grasping.

Ultimately, this work provides a "drop-in" diagnostic tool for engineers. Instead of just chasing higher accuracy percentages, developers of assistive systems can use $U_{id}$ to ensure their decoders aren't leaving users stuck in a "pathological" confusion that prevents them from ever reaching their goal.

Summary updated

Abstract

A central challenge in assistive brain-to-robot control is decoding a user's latent intent from complex and noisy brain signals. Prior approaches evaluate decoders as single-trial classifiers with accuracy-based metrics, but this neglects the inherently sequential nature of assistive control. Since a planner must integrate intermittent, noisy decoder outputs over time, single-step accuracy can only be a proxy when predicting downstream task performance. We formalize brain-to-robot control as a partially observable Markov decision process (POMDP), and prove that there is a sharp phase transition: regret is uniformly bounded across the horizon as long as pairs of goals are all decoder-distinguishable, but grows linearly otherwise. We show that a single channel statistic, the worst-pair Chernoff information $Γ_{\min}$ sets this boundary. Then, we derive a closed-form regret predictor $U_{\mathrm{id}}(C)$, that can be computed from any confusion matrix in $O(K^2)$ time. Across two domains spanning orthogonal information channels, we show that $U_{\mathrm{id}}$ predicts cumulative planning regret beyond what accuracy can explain. We then highlight a pathological accuracy trap, in which two decoders have identical validation accuracy 35.1% on real EEG data (BNCI2014001 subject 5) but result in a difference of $9.6\times$ in held-out regret, which $U_{\mathrm{id}}$ predicts. Same-signal control confirms $U_{\mathrm{id}}$'s advantage with decoders that span independent feature channels. Ultimately, our work provides a drop-in diagnostic to evaluate and select sensors and decoders in sequential assistive control.
BibTeX
@misc{kim2026beyond,
  author = {Tasha Kim and Benjamin Ballyk and Anson Lei and Ingmar Posner and Oiwi Parker Jones},
  title = {{Beyond Accuracy: Information-Theoretic Decoder Evaluation Under Latent Intent}},
  year = {2026},
  eprint = {2611.05658},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05658},
}
Figure