Scheduling Robot Attention When Listening Is Self-Defeating

A robot that keeps track of several people talking around it faces a familiar problem from wireless scheduling. It can only attend closely to one source at a time, and its estimate of everyone else goes stale. Wireless schedulers handle this with restless bandits, a model of many targets that keep changing whether or not they are watched, and with index policies that score each target by how valuable it is to look at right now. A robot adds a twist that those schedulers never model. To look somewhere, it has to move: turn its head, rotate its base, walk. Moving runs motors, fans and feet, and that noise lands in its own microphones. This "ego-noise" can triple wake-word false rejections and push localization error up by tens of degrees (Alpha-mini challenge). The act of exploring damages the observation that exploring was supposed to produce, and it degrades what the robot hears from every other source too.

We propose to model the robot's own noise as a shared, priced resource inside a restless bandit. In each time slot the robot picks a motion mode: stand still, move quietly, or move fast. It also picks which sound source to attend to. A second Lagrange multiplier prices the noise that the chosen motion injects into all channels at once. The resulting policy answers two questions together that current systems answer separately or not at all: when to stop and listen, and to whom. We will check it against exact dynamic programming on small instances, in a simulated home whose robot is made audibly noisy, and against the hand-designed and learned alternatives a reviewer would reach for first.

Core Claim

Restless-bandit scheduling assumes that probing a target yields a clean observation and leaves the other targets untouched. Robotic ego-noise breaks both assumptions. Observation quality depends on the motion that achieves attention, and that dependence couples every arm (each target being tracked) through one global noise level. Our claim is that this coupling can be relaxed with a second price. Each arm then gets an index that depends on the current motion mode, the robot picks the motion mode that maximizes task progress plus the top index, and the policy stays near-optimal while running in linear time.

The prediction is concrete and falsifiable. The policy should beat a noise-blind index scheduler and a fixed "stop and listen every k seconds" rule on the combined objective of tracking uncertainty and travel time. It should also stay within a few percent of the exact optimum where the optimum can be computed.

Model

Each of $M$ sources (speakers or sound events) is a two-state Markov process, active or silent, as in Uncertainty-of-Information Scheduling. That paper scored freshness as the entropy of the robot's belief, and it showed that this entropy can oscillate rather than grow. That property matters for bursty conversation, where a person who has gone quiet may not be worth re-checking soon.

We depart from that paper in how observations work. The robot holds a belief $\pi_i(t)$, its probability that source $i$ is active. Observing source $i$ does not collapse this belief to 0 or 1. Instead, the robot receives a detector output with a confusion matrix $C(q)$, where $q$ is the effective signal-to-noise level. That level depends on two things: whether source $i$ is the attended one (beamformed or faced) and the motion mode $a_t$. Unattended sources are still heard, just through a worse $C$. We calibrate $C$ empirically for each pair of motion mode and attention state from recorded ego-noise. The candidate noise sources are the Alpha-mini joint-noise subset, NAO arm-motion noise from Partially Adaptive Multichannel Joint Reduction, and gait noise levels like the 65.78 versus 79.74 dBA contrast reported for MUTE.

Let $H(\pi)$ be the binary entropy of a belief $\pi$, $r(a)$ the per-slot task progress of motion mode $a$, and $\gamma$ a weight trading uncertainty against progress. The robot minimizes long-run average cost under a one-source attention constraint. Relaxing that constraint with a subsidy $\lambda$, and the coupling through the shared mode with a noise price $\nu(a)$, gives single-arm problems:

$$\min \; \lim_{T\to\infty}\frac{1}{T}\,\mathbb{E}\sum_{t=1}^{T}\Big[H(\pi_i(t)) - \lambda\,(1-u_i(t)) + \nu(a_t)\Big]$$

Here $u_i(t)\in\{0,1\}$ records whether source $i$ is attended at time $t$. Each single-arm problem has a one-dimensional belief state, so its index $W_i(\pi;a)$ is computable by value iteration over a discretized belief. At run time the robot evaluates, for each mode $a$, the quantity $\gamma r(a) + \max_i W_i(\pi_i;a)$ and acts on the best one.

Indexability is not guaranteed once observations are noisy. The UoI paper's proof relies on the belief collapsing after each observation. We will test indexability numerically for each calibrated $C$. Where it fails, we will report the failure and fall back to a computed priority from the same single-arm value functions.

Research Program

The work has three parts.

The first is calibration. Using the Alpha-mini development conditions, which range from keyword-only to keyword plus echo plus mechanical noise, together with the NAO recordings, we estimate voice-activity and keyword-detection confusion matrices for each motion mode. These numbers ground the observation model in measured degradation rather than assumed noise. The Alpha-mini paper reports localization accuracy of about 46% with noise and about 29% with echo plus noise, which already shows a steep drop as noise conditions worsen.

The second is theory and verification on small instances. For $M \le 5$, exact relative value iteration over the joint belief is tractable. This follows the verification protocol of the UoI paper, whose Whittle policy came within 0.1% of the optimum while a myopic policy lost more than 20%. We will measure how much of that near-optimality survives once observations become noise-coupled.

The third is embodied evaluation. SoundSpaces 2.0 renders sound reaching a moving listener. Like the agent in Active Audio-Visual Separation, its simulated agent is silent. We will add the robot's own noise to the microphone signal: recorded ego-noise clips, mixed at levels set by the active motion mode and turning speed, ahead of a fixed voice-activity and keyword detector. The task is to traverse a home to a goal while catching wake words from $M$ intermittently talking occupants.

The design also borrows from active audition. Head rotation resolves front-back confusion (Ma et al.), and closed-loop steering beats periodic scanning (Schymura et al.). We keep their insight that motion is informative, and add the cost they ignore: the motion is loud. Quiet-gait controllers such as MUTE and QuietPaw supply the motion modes. We treat their quietness setting as something the scheduler chooses rather than an operator's fixed dial. Their reported speed-versus-quietness tradeoff is exactly why choosing that setting dynamically could pay off.

Baselines

The naive reference is a greedy scheduler. It attends to whichever source currently looks most likely active, moves at full speed, and treats the noise as just bad luck. The UoI paper already reports greedy regret above 20% in the noise-free case, so we expect it to set the floor.

The strongest structured competitor is the noise-blind Whittle index policy from Uncertainty-of-Information Scheduling. Its index is computed as if every observation were perfect, and motion mode is chosen independently by a goal-reaching controller. Its reported result is about 0.1% from optimal on average sum-UoI without ego-noise. Any gain we show over it is attributable to modeling the noise, because both policies use identical beliefs, detectors and environments.

The practical engineering competitor is periodic stop-and-listen. The robot halts every $k$ slots, attends to sources in round-robin order, then resumes at full speed. We tune $k$ and the attention order by grid search under the same simulation budget we give our own hyperparameters.

The learned competitor is a PPO agent in the style of Majumder and Grauman. It sees the same audio features and beliefs, it trains in the same noisy simulator, and it has a matched environment-step budget. We found no reported result on Lacuna for any policy trained with agent ego-noise.

Three further references isolate parts of the method. A static-stealth baseline always uses the quietest mode, to show whether lost progress outweighs cleaner hearing. An ablation keeps the noise-aware confusion matrices but drops the noise price $\nu$, which isolates the effect of coupling all arms through one shared noise level. For small $M$, exact relative value iteration serves as the ceiling.

Beating a baseline means a lower combined cost (entropy plus $\gamma$ times travel time) across a sweep of $\gamma$, i.e., a dominating Pareto curve. At the operating point, it also means a lower missed-wake-word rate at equal or shorter travel time, over at least 20 scene seeds with confidence intervals.

Why not just…?

Why not just stop and listen periodically? This is the most serious threat. Halting to listen captures much of the benefit with no theory at all. The difference is that a fixed period ignores that sources differ in how bursty they are. Sources whose entropy oscillates rarely need re-checking, while fast-switching sources need it often. Schymura et al. found closed-loop steering beats periodic scanning even without modeling noise. If a tuned stop-and-listen rule matches our policy within confidence intervals across mixed speaker dynamics, we will report that the structure is unnecessary for this task.

Why not just denoise better and keep the noise-blind scheduler? Front-end processing helps but falls far short. Fully adaptive multichannel factorization pushes word error above 0.85, against 0.65–0.70 when the ego-noise model is fixed (Partially Adaptive Multichannel Joint Reduction). Multichannel averaging on RoboVox still leaves 15.06% equal error rate against 1.12% clean (RoboVox). We will run the noise-blind index policy with the best available front end. Any residual degradation enters our model through $C(q)$, so better denoising and our scheduler are complementary. The experiment measures how much scheduling gain remains after denoising.

Why not just train an end-to-end policy in a noisy simulator? It might work. However, it gives no optimality certificate, has to be retrained whenever the number of sources or the robot's noise profile changes, and its sample cost grows with $M$. Our index policy is recomputed per arm in linear time. The matched-budget PPO comparison, plus a transfer test to unseen values of $M$ and unseen noise profiles, settles the question.

Decisive Checks

Three results would establish the claim. First, on small instances, the noise-aware index should come within a few percent of relative value iteration, while the noise-blind index's gap should grow as ego-noise worsens. Second, in the noisy SoundSpaces homes, our policy's Pareto curve should dominate both tuned stop-and-listen and the no-price ablation. Third, it should transfer to new values of $M$ without loss, where PPO degrades.

If the no-price ablation matches the full method, the shared-resource coupling is not what matters, and we would narrow the contribution to the noisy observation model. If tuned stop-and-listen matches the full method, the honest conclusion is that a simple rule suffices for this regime.