Deciding When to Aim: Deadline-Aware Commitment for Steered Speech Extraction

A microphone array can pull one voice out of a crowded room by "aiming" at the talker. It delays and combines the microphone signals so that sound from one direction adds up and sound from other directions cancels. Aiming only works if the system knows the talker's direction of arrival (DOA), and aiming slightly wrong is very costly. Neural spatial filters lose much of their ability to isolate a talker when the angle is off by as little as 2°. When a new person starts talking, the system faces a timing problem. If it aims early, it uses a noisy angle estimate and may lock onto the wrong spot. If it waits for a precise estimate, it falls back on weaker, unaimed processing during the very words it most needs to capture, and the speaker turn may end before it ever commits. We propose to treat the moment of committing to a direction as an optimal stopping problem with a deadline. The cost of stopping is the expected loss in extraction quality from aiming with the current angle uncertainty. The cost of waiting is the quality lost by staying in a safe, unaimed mode. The deadline is the uncertain end of the turn.

Decision theory predicts the shape of the right policy for problems like this. When the deadline is uncertain, the evidence required before committing should shrink as time passes, a so-called "collapsing boundary." That lets the system commit quickly on easy turn onsets, wait on ambiguous ones, and still commit before the turn runs out. The new element is that the boundary is learned against downstream extraction quality, which differs between extractors, rather than against angle accuracy alone.

Core Claim

For target-speaker extraction at turn onsets with unknown talker location, a learned, deadline-aware collapsing boundary on the DOA posterior should beat the obvious fixed policies. Those fixed policies are "always wait N ms," "commit at a fixed confidence," and "steer continuously at the running estimate." We will compare them on the curve of extraction quality against commitment latency. The gains should concentrate where onset difficulty varies most: closely spaced talkers, low signal-to-interference ratio, and high reverberation. If the gains are uniform or absent, the claim is wrong. In that case a well-tuned fixed wait is sufficient, and we will report that.

Why This Is Not Already Done

Prior work on the spatial side either takes the direction as given or uses fixed-length inference. L-SpEx (L-SpEx: Localized Target Speaker Extraction) estimates the target's DOA internally from a reference voice sample and feeds it to an aiming beamformer. It does so in one forward pass over a fixed window and never decides when its angle is good enough. The Directional Recurrent Network (All Neural Low-latency Directional Speech Extraction) and FiLM-conditioned hearing-aid extraction (Location-Aware Target Speaker Extraction for Hearing Aids) both accept the DOA as an external per-frame input. Kim's conic voice activity detector (Multi-channel voice activity detection based on conic constraints) tolerates angle error through a fixed geometric acceptance cone, a spatial gate with no notion of time.

On the decision side, the relevant methods have never been applied to spatial audio. FIRMBOUND (Learning the Optimal Stopping for Early Classification within Finite Horizons via Sequential Probability Ratio Test) learns finite-horizon stopping boundaries from data, but only on vision and sensor series. DDDM (DDDM: A Brain-Inspired Framework for Robust Classification) accumulates stochastic evidence until it reaches a fixed 0.99 threshold, and only on single-channel keyword classification.

The obvious combination would apply FIRMBOUND to DOA classification and score the result by angle error. We do something different in two ways. First, the cost of stopping is the extractor-specific sensitivity of extraction quality to angle error. Second, the alternative to committing is a working fallback extractor, not silence.

Making the cost extractor-specific matters, and the file shows why. Tesch and Gerkmann report that 2° errors substantially impair their steerable filter (Multi-channel Speech Separation Using Spatially Selective Deep Non-linear Filters). DRN, by contrast, is flat up to ±5° of mismatch. These results appear to conflict. We read them as evidence that the right stopping rule depends on the extractor, which a rule based only on angle accuracy cannot capture.

Research Program

The two operating modes. The system has a safe mode: L-SpEx-style extraction that uses the speaker embedding and treats the internally estimated DOA only as a soft feature. It also has a committed mode: a spatially selective steered filter in the style of JNF-SSF (Multi-channel Speech Separation Using Spatially Selective Deep Non-linear Filters). The committed mode is conditioned on a point DOA through exponential encoding and FiLM, following Location-Aware Target Speaker Extraction for Hearing Aids.

The evidence stream. A frame-wise DOA posterior $p_t(\theta)$ over an azimuth grid is accumulated causally. We will produce it with a sequential density-ratio estimator of the SPRT-TANDEM type used inside FIRMBOUND, because that estimator handles evidence frames that are correlated in time. That matters here because the stopping theory of Frazier et al. and of Huang and Rao assumes independent observations, which reverberant audio frames are not.

Measuring each extractor's sensitivity. Offline, we measure a sensitivity curve $\ell(\Delta)$ for the committed extractor. This curve is the expected per-frame drop in scale-invariant signal-to-distortion ratio (SI-SDR, a standard separation-quality score) when the extractor is steered $\Delta$ degrees off the true angle.

Stopping and waiting costs. The risk of committing at frame $t$ to the posterior mode $\hat\theta_t$ is

$$R_{\text{stop}}(t) = \mathbb{E}_{\theta \sim p_t}\big[\ell(|\hat\theta_t - \theta|)\big] \cdot (T - t),$$

where $T$ is the turn end, so the loss is paid on every remaining frame. Waiting one more frame costs $c(t)$. This is the measured per-frame quality gap between safe mode and oracle-steered mode, plus an expectation over a stochastic deadline: if the turn ends before commitment, the committed mode's benefit is forfeited.

The deadline model. The deadline distribution is fit to turn-duration statistics. For the log-concave distributions we expect, such as Gamma, Frazier et al. (Sequential Hypothesis Testing under Stochastic Deadlines) prove that the region in which the system keeps waiting shrinks monotonically over time. Our learned boundaries should reproduce this shape, and that serves as a sanity check.

Learning the boundary. We solve the backward induction offline with FIRMBOUND's Gaussian-process variant, which the paper reports trains 30x faster than its convex-function-learning variant. The input is a two-dimensional state: elapsed time and posterior concentration, the latter measured as the log-ratio between the top two angle hypotheses. Online, the stopping test reduces to a lookup against a precomputed curve, adding negligible cost per frame.

Re-aiming after commitment. After committing, a lighter version of the same test, run with a "decreasing optimism" schedule as in When to Make and Break Commitments?, decides whether to drop the commitment when the posterior drifts. This step is secondary. We include it only so that committing early is not artificially penalized for being irreversible.

Baselines

The naive reference points run on identical causal mixtures. The first never commits: it runs L-SpEx alone, which reports 9.68 dB SDR on MC-Libri2Mix and 8.97 dB for talkers separated by less than 45° (L-SpEx: Localized Target Speaker Extraction). The second commits immediately on the first-frame DOA estimate. Our method must beat both. If it cannot beat never-committing, steering is not worth the risk in this regime. The oracle upper bound steers the committed extractor at the true DOA from frame one. For context on the steered filter's reported strength, JNF-SSF reports 15.1 dB SI-SDR improvement with five speakers in its own pyroomacoustics setup. That is a different benchmark, so it is not directly comparable.

The fixed-wait policy is the strongest simple competitor. It accumulates the DOA posterior for exactly N ms, then commits. We sweep N over a fine grid on the same validation data and give it the same tuning budget as our boundary learner, which gives a full quality-against-latency curve. Beating it means our method sits strictly above that curve at matched mean commitment latency, with a confidence interval that excludes zero, by bootstrapping over test turns.

The static-threshold sequential test is a Wald-style rule. It commits when the top-two log-ratio crosses a constant level, with a forced commit at a timeout. Both the level and the timeout are tuned jointly on validation data. It isolates the value of the boundary collapsing over time.

The continuous soft steering baseline never makes a discrete decision. It steers the committed extractor at the running posterior mean every frame, in the spirit of DRN's frame-wise DOA conditioning (All Neural Low-latency Directional Speech Extraction). We train it with angle-mismatch augmentation so it is as tolerant to error as possible. DRN reports STOI 83.2 and 4.98 dB SI-SDR on its DNS-based setup. Lacuna lists no reported result for DRN on MC-Libri2Mix, so we will run it ourselves.

The key ablation keeps our entire pipeline but replaces the sensitivity curve $\ell(\Delta)$ with plain angle error as the stopping cost. This is FIRMBOUND applied naively to DOA. If this ablation matches the full method, the loss-aware ingredient adds nothing. A second ablation removes the deadline model and uses a constant waiting cost.

On published results for adaptive commitment, Lacuna has no reported result for any adaptive-commitment DOA policy on these benchmarks.

Why not just…?

Why not just wait a fixed, well-tuned amount of time and then commit? This is the most serious threat. If most turn onsets are equally hard, an adaptive rule cannot beat a tuned constant. Our case rests on heterogeneity of onset difficulty. The file shows that quality depends strongly on angular separation, with L-SpEx dropping in the under-45° band and spatial filters degrading as interferers approach the target. FIRMBOUND also reports that adaptive stopping reduces decision-time variance and defines the speed-accuracy Pareto front against fixed-horizon models. Neither fact proves the effect exists in audio. The experiment that settles it stratifies results by angular separation and signal-to-interference ratio. We will also construct a deliberately homogeneous test condition in which we predict no gain. If the gain does not appear in the heterogeneous strata, we will conclude fixed-wait suffices.

Why not just use a constant confidence threshold, Wald-style? Wald's test is optimal only with an unbounded horizon and constant costs. Frazier et al. prove that constant boundaries are suboptimal under stochastic deadlines. Drugowitsch et al. (The Cost of Accumulating Evidence in Perceptual Decision Making) find that time-varying boundaries fit primate and human behavior better than fixed ones. The threat is practical rather than theoretical. If typical commitment times are short relative to turn length, the deadline rarely binds, and a constant threshold plus a timeout behaves almost identically. We test this directly by varying the turn-length distribution. The collapsing boundary's advantage should grow as turns get shorter and more variable, and it should vanish for long turns.

Why commit at all, rather than steer softly at the running estimate? Steering continuously avoids a hard decision. However, a steerable filter's error is paid on every frame. Early posterior means are noisy, and Tesch and Gerkmann show that errors around 2° already hurt a selective filter. A wide safe mode loses less than a narrow beam aimed slightly wrong. This argument depends on the extractor, though, and DRN's flatness up to ±5° suggests a mismatch-tolerant extractor might make commitment unnecessary. We therefore run soft steering with mismatch augmentation as a full baseline. If it matches our method, the honest conclusion is that robust conditioning beats deciding when to aim, and we will report it that way.

Proposed Experiments

Data and setting. The primary testbed is MC-Libri2Mix, a four-microphone linear array with reverberation times of 200–600 ms. We will re-segment it into causal multi-turn streams in which a new target begins talking at a random time and location, with turn durations drawn from Gamma distributions of varying spread.

Measuring the sensitivity curve. To measure $\ell(\Delta)$ under denser conditions, we reproduce the Tesch and Gerkmann pyroomacoustics setup: a three-microphone, 10 cm circular array with up to five speakers and MUSAN music interference. We will also measure the curve for a DRN-style extractor, which tests our premise that the optimal boundary depends on the extractor.

Metrics. We report per-turn SI-SDR and SDR, quality restricted to the first second of each turn (where commitment matters), mean and variance of commitment latency, and the rate of wrong-direction locks. We also report word error rate from an off-the-shelf recognizer. We include it because Spille et al. (Identifying the human-machine differences in complex binaural scenes: what can be learned from our auditory system) attribute much of the gap between humans and machines in speech recognition to localization and tracking errors.

Decisive checks. Four results would decide the question. The first is Pareto dominance over the tuned fixed-wait curve, concentrated in the under-45° and low signal-to-interference strata. The second is the loss-aware ablation falling below the full method; this separates our proposal from naive FIRMBOUND-on-DOA. The third is learned boundaries that collapse earlier as deadline uncertainty grows, as Frazier et al. predict. The fourth is a domain-shift test: we perturb array geometry by 1 cm, the displacement Tesch and Gerkmann found damaging. We then measure how much of the learned boundary's advantage survives. We also test whether re-measuring only $\ell(\Delta)$, without retraining the DOA estimator, restores it. This addresses FIRMBOUND's stated assumption that training and test data come from the same distribution.