Motor-Aware Abstention: Risk-Controlled Robot Hearing Under Self-Generated Noise

A robot's microphones sit a few centimeters from its own motors, fans and gearboxes. When the robot moves, that self-generated noise, called ego-noise, can bury the speech the robot is trying to hear. Today's robot listeners still output an answer on every input. A better robot would say "I didn't catch that" when its own body was too loud, and it would do so at a rate the designer can bound. Selective prediction is the standard tool for this. It pairs a predictor with a gate that refuses to answer on some inputs, and it can certify that the error rate on the inputs it does answer stays below a chosen budget. That certificate has two weak points. It holds only if the gate was tuned on data drawn from the same conditions the robot will face, and it was developed for static images and text, not for a sensor whose noise source is the device itself.

This proposal rests on one observation. Ego-noise is the one kind of interference a robot can both predict and record on its own. The robot knows its joint commands, and it can record its own motor noise in a silent room with no human labeling at all. We propose a gate that takes the robot's motor state as an input alongside the audio. Its error guarantee is stated per motion state: for example, at most 2% false wake-word acceptances while walking and while standing, with coverage allowed to differ between the two. The gate is tuned on hybrid data that mixes real recorded ego-noise with labeled speech, which avoids collecting labeled speech-in-motion on the real robot. The central empirical question is whether such a guarantee survives the gap between simulation and reality. That gap concerns room acoustics and how motor vibration couples into the microphones. Where the guarantee breaks, we also want to measure by how much.

Core Claim

We claim that motor-state conditioning and real-ego-noise calibration together yield abstention that stays within its error budget on fully real robot recordings. A gate calibrated on purely simulated noise does not. We further claim this abstention answers more often than the obvious rule of "refuse whenever motors are active." A secondary claim is that training the predictor and gate together closes part of the gap between simulated and real recordings. In this joint training, each training clip's loss is weighted by the gate's admission probability. The effect is to reduce over-abstention, the failure mode in which a noisy model refuses even inputs it would have gotten right.

Background and Positioning

Selective prediction supplies the method. Selective Classification for Deep Neural Networks introduced Selection with Guaranteed Risk (SGR). SGR thresholds a confidence score, such as the maximum softmax probability, using a held-out calibration set and a binomial bound. The result is an error-rate guarantee on admitted inputs that holds with probability $1-\delta$. That paper reports 1% error at 78% coverage on CIFAR-10. It also shows that softmax confidence ranks inputs about as well as Monte Carlo dropout, which is why we use softmax confidence as the base score. SGR assumes calibration and test data come from the same distribution, and it tunes the gate only after the predictor is trained. Both assumptions are what break on a robot.

Selective "Selective Prediction" supplies the motivating failure. It shows that standard gates on vision-language models abstain on 96% of queries to reach 90% accuracy. Its fix, ReCoVERR, decomposes questions into simpler sub-questions. That remedy is specific to language and has no counterpart for a single noisy audio frame. We take the diagnosis, a blanket loss of confidence under noise, and attack it through motor state instead.

The robot-audition papers establish the problem and constrain the design. The Alpha-mini Speech Challenge shows three things on a humanoid. Keyword-spotting false rejects triple once real servo noise and echo are added. Localization accuracy within 10° falls from about 46% to about 29%. Simulated echo fails to match the real hardware. RoboVox shows speaker-verification equal error rate rising from 9.29% on a lapel microphone to 15.63–18.22% on the robot's own microphones. Usefully, RoboVox includes an internal chassis microphone, which can serve as a direct reference for ego-noise.

None of these baselines abstains. Partially Adaptive Multichannel Joint Reduction of Ego-noise and Environmental Noise supplies an enabling fact. It learns motor-noise models from isolated joint recordings, which shows that ego-noise can be captured offline and tied to posture. Even so, its speech recognizer's word error rate stays around 0.65–0.70, so denoising alone leaves much of the error in place. ManiWAV overlays recorded motor noise onto clean demonstrations. That is the same additive-mixing approximation our calibration relies on. ManiWAV uses it only for augmentation, without testing whether it is faithful enough to support a guarantee.

On the transfer side, RealMAN quantifies the acoustic gap between simulation and reality: localization accuracy within 5° drops from 89.2% when training on real recordings to 52.4% when training on simulation. Its result warns us that simulated room responses alone should not be trusted for calibration. This is why we propose real recorded ego-noise rather than simulated ego-noise.

Two papers supply the weighted-loss technique. Rectifying Pseudo Label Learning via Uncertainty Estimation weights each training example by an uncertainty estimate and adds a penalty that stops the model from declaring everything uncertain. That reaches 50.3% versus 48.4% mean intersection-over-union on GTA5→Cityscapes. Bootstrapping single-channel source separation shows that confidence-weighted training helps in audio, gaining 1.1 dB in signal-to-distortion ratio. Both use their weights only at training time and never abstain at deployment. Our version makes the training weight and the deployment gate the same function.

Research Program

The input is an audio window $x$ together with a motor-state descriptor $m$. In the simplest case, $m$ contains joint velocities and fan state from the robot's own logs. If those logs are unavailable, an internal-microphone energy feature stands in. The predictor $f(x,m)$ is a standard keyword-spotting, speaker-verification or localization network. The gate $g(x,m)\in[0,1]$ is a small head on shared features that outputs a probability of admission.

We discretize motion into a few states $s(m)$, such as standing, arm motion and walking. The guarantee we target is conditional on that state:

$$\Pr\Big[\, R_s(f,g) \le r^* \ \text{for every motion state } s \,\Big] \ge 1-\delta,$$

Here $R_s(f,g)$ is the error rate over admitted inputs while the robot is in state $s$. The budget $r^*$ is user-set and $\delta$ is the allowed failure probability. We obtain this guarantee by running SGR separately within each state and applying a union bound across states.

Training is joint. For a training set of $N$ clips, we minimize

$$\frac{\sum_{i=1}^{N} g(x_i,m_i)\,\ell(f(x_i,m_i),y_i)}{\sum_{i=1}^{N} g(x_i,m_i)} + \lambda\big(c - \tfrac{1}{N}\sum_{i=1}^{N} g(x_i,m_i)\big)_+^2,$$

where $\ell$ is the task loss, $y_i$ the label, $c$ a target coverage (fraction of inputs answered) and $\lambda$ the penalty weight. The second term plays the same role as the regularizer in Rectifying Pseudo Label Learning via Uncertainty Estimation: it stops the gate from rejecting everything. With this objective, the predictor stops spending capacity on clips that are hopeless at their ego-noise level. After training, the thresholds are still set by SGR on held-out calibration data, so the guarantee does not depend on the training objective.

The data plan has three tiers, all on one physical robot. First, we record ego-noise only, with synchronized joint logs, across scripted motion primitives in a quiet room. This needs no labels and can scale to hours. Second, we build hybrid mixtures for training and calibration by adding these recordings to labeled speech. The speech is either played through a loudspeaker or convolved with simulated room responses such as those in VAST. Third, we hold out a fully real test set, recorded with people speaking while the robot actually moves. This tier is used only for evaluation and for an oracle calibration.

The comparison across calibration sources is the scientific core. It separates three effects. One is the cost of simulated noise, from calibrating on fully simulated mixtures. Another is the cost of additive mixing, from calibrating on hybrid mixtures. The third is the irreducible gap, measured against an oracle calibrated on real data.

Baselines

The naive reference point is the unconditional predictor at full coverage, exactly as the Alpha-mini and RoboVox baselines are run. On RoboVox the robot-microphone equal error rate is 15.63–18.22%, and averaging channels gives 15.06% (RoboVox). On Alpha-mini we know only that false rejects triple under mechanical noise, with no absolute value given (Alpha-mini Speech Challenge). We found no reported selective-prediction result on Lacuna for Alpha-mini, RoboVox, the Robot Kidnapper dataset or RealMAN. The closest reported number is SGR's 1% error at 78% coverage on CIFAR-10 (Selective Classification for Deep Neural Networks). That figure is not comparable to our tasks and serves only as a sanity anchor, so we will establish all baseline numbers ourselves.

The strongest existing method is softmax-confidence SGR applied to the same predictor architecture. It will be trained with uniform loss on the identical hybrid training data, with the same compute and the same hyperparameter search budget, and calibrated three ways: on fully simulated mixtures, on hybrid mixtures, and on real data. Our method beats it if two things hold on the real test set. Its guarantee-violation rate must stay at or below $\delta$ where simulation-calibrated SGR exceeds it. And at the same error budget, it must answer at least five percentage points more inputs than hybrid-calibrated SGR, averaged over motion states.

The motor-state gate is the simplest proprioceptive competitor. It refuses whenever the robot is in a high-noise motion state and otherwise applies SGR. The oracle is SGR calibrated on real labeled data from each motion state; it bounds what any calibration source could achieve.

The ablations isolate our ingredients. Removing $m$ from the gate tests motor conditioning. Replacing joint training with a post-hoc gate on a uniformly trained predictor tests the weighted loss. A front-end ablation places the partially adaptive denoiser of Partially Adaptive Multichannel Joint Reduction of Ego-noise and Environmental Noise ahead of SGR, testing whether denoising simply makes abstention unnecessary.

Why not just…?

Why not just calibrate SGR on real labeled recordings? That is the oracle, and it is valid. The problem is cost. It needs labeled speech per motion state, per room and per hardware revision, and this multiplies as the robot gains behaviors. We will measure how many real labeled clips per state are needed before oracle calibration matches hybrid calibration. If the answer is a few dozen, our contribution shrinks to a convenience, and we will say so.

Why not just refuse whenever the motors are running? This is the most serious threat, because it is cheap and probably safe. Our concern is that it throws away the many moving-state inputs that are still answerable, which is the over-abstention pattern documented by Selective "Selective Prediction". The comparison of coverage at a fixed error budget settles this directly. If the learned gate does not beat the motor-state rule in moving states, the honest conclusion is that proprioception, not learning, does the work. That would still be a useful negative result.

Why not just denoise first and keep the predictor unconditional? Denoising helps but leaves a large error floor. The partially adaptive denoiser still yields word error rates of 0.65–0.70 (Partially Adaptive Multichannel Joint Reduction of Ego-noise and Environmental Noise). Quiet locomotion leaves a fan-and-electronics floor of about 55 dBA (Minimizing Acoustic Noise). Denoising also offers no error guarantee. We therefore treat it as complementary and test the denoiser-plus-SGR combination as a baseline.

Why not just use softmax confidence alone? Softmax confidence ranks well on clean data (Selective Classification for Deep Neural Networks). Under noise the network has not seen, however, it can be confidently wrong, and the motor-state input exists precisely to warn the gate in those cases. The ablation without $m$ measures this directly.

Proposed Experiments

We evaluate keyword spotting on the Alpha-mini data, where real mechanical-noise recordings exist (Alpha-mini Speech Challenge). We evaluate speaker verification on RoboVox, using the internal chassis microphone channel as the motor-state signal (RoboVox). Localization is evaluated on the Robot Kidnapper dataset, whose split into static and moving robot recordings gives two motion states, with a mean absolute error baseline of 21° static and 26° moving (The Un-Kidnappable Robot). RealMAN is used to measure how simulated room responses distort calibration (RealMAN). Public datasets may lack joint logs. In those cases the internal-microphone or static/moving labels stand in for $m$, and our own robot collection supplies the true proprioceptive version.

The metrics are four. First, realized error at each coverage level. Second, area under the risk–coverage curve. Third, coverage at error budgets of 1%, 2% and 5%. Fourth, the guarantee-violation rate: the fraction of 200 bootstrap test draws in which realized risk in some motion state exceeds $r^*$. All are reported per motion state.

Three results would be decisive. The first concerns validity. Simulation-calibrated SGR should violate its guarantee in moving states more often than $\delta$, while hybrid calibration should not. If hybrid calibration also violates it, we will report the slack, meaning how much the budget must be loosened to restore validity, as the measured price of the gap between simulation and reality. The second concerns usefulness. The learned motor-aware gate should beat the motor-state rule on coverage at equal error budget in moving states. The third concerns training. Joint weighted training should improve coverage at fixed error over a post-hoc gate on a uniformly trained predictor, with motion state held fixed. If only the first result holds, the paper's contribution narrows to a calibration recipe, and it will be framed that way.