A microphone array can locate someone only while they are making sound. When a talker walking through an office stops speaking, every acoustic tracker goes blind, and the gap is filled by guesswork. Current trackers guess forward in time. They run a filter that carries the last estimate ahead with a simple motion model until speech resumes. When speech does resume, the filter has usually drifted, and nothing goes back to correct the silent stretch. That matters for any system that must know where the speaker was, such as a beamformer that has to line up its spatial filter with the true source path, a diarization log, or a robot's map of who went where.
This note proposes to reconstruct the silent stretch after the fact, using both the point where the talker fell silent and the point where they spoke again. The reconstruction is a probability distribution over routes, not a single interpolated line, because two anchors on opposite sides of a pillar or partition can be joined by more than one route. The acoustic evidence at both anchors is judged with a model of room reflections, so echoes are not mistaken for positions. The claim is concrete: in rooms with obstacles, a mixture of smoothers indexed by route, with a likelihood that explains echoes, recovers where the talker actually walked, including which side of an obstacle they passed. We expect it to do this better than forward trackers, better than deterministic bidirectional imputers, and, most importantly, better than a well-tuned forward-backward particle smoother.
Core Claim
Three physical facts combine into one estimator. First, a silent interval is bounded by two acoustic anchors, and anchoring at both ends stops the error from compounding. Multiagent off-screen behavior prediction in football shows this with its Graph Imputer, a bidirectional recurrent network with a variational autoencoder that fills in players' off-camera paths. That work runs on clean optical coordinates, where an agent is either perfectly observed or missing. Second, the route between the anchors is often a discrete choice. Multiple Futures Prediction shows that a squared-error model facing two options predicts their average, which here means a path through the obstacle. Third, the anchors are not equally trustworthy. The direct sound always arrives before its reflections. The first frames of a new utterance (the onset) are therefore dominated by the direct path. The last frames before silence (the offset) are smeared by the reverberant tail of everything said before. We therefore predict that the reappearance anchor is systematically more reliable than the disappearance anchor, and that a good estimator should weight them unequally. No imputation method in the file models this asymmetry. BRITS, the standard bidirectional recurrent imputer, explicitly treats present observations as reliable.
The estimator writes the posterior over the gap trajectory $x_{a:b}$, which runs from offset time $a$ to onset time $b$ given acoustic frames $y$, as a mixture over routes $h$. Each route is a topologically distinct way around the obstacles in a known floor plan:
$$ p(x_{a:b}\mid y) = \sum_h w_h \, p(x_{a:b}\mid y, h), \qquad w_h \propto p(h)\, p(y_{\le a}, y_{\ge b}, \tau \mid h), $$
Here $p(h)$ is a route prior, and $\tau = b - a$ is the gap duration. The duration term carries real information even in total silence: a talker walking at normal speed cannot take the long way around in two seconds. Each component $p(x_{a:b}\mid y, h)$ is a kinematic smoother kept inside its route's corridor. Keeping the route explicit as a mixture stops a single set of particles from collapsing onto whichever route happened to get lucky early.
Where This Sits in Prior Work
The trajectory-forecasting literature supplies the representation. MultiPath represents futures as a set of anchor trajectories with offsets. Uncertainty-Aware Driver Trajectory Prediction at Urban Intersections uses Gaussian mixtures to put probability on separate maneuvers and near-zero probability between them. Trajectron++ enforces kinematic feasibility by predicting controls and rolling them through a vehicle model. HOME avoids choosing a mixture size in advance by predicting heatmaps. All four only forecast forward from visual or map context, and none has an acoustic observation channel. We borrow the mixture-over-discrete-routes idea and the kinematic rollout, and apply them to a bounded gap rather than an open future.
The imputation literature supplies the two-anchor structure. BRITS ties its forward and backward passes together with a consistency loss, but it outputs a single deterministic estimate. The Graph Imputer is generative, but it has no notion of corrupted observations. BCDiff, ProDiff and AISFlow cast gap filling as denoising or flow matching conditioned on the endpoints. These methods could represent multiple routes, but they need large trajectory corpora and are not tied to a room's geometry. MS-TIP imputes randomly masked pedestrian histories, but without any physical model of why data went missing. Our contrast with all of them is the same. Missingness here has a physical cause, silence, and the frames around it carry echoes that must be explained, not trusted.
The acoustic literature both motivates and constrains the design. The motivation comes from The LOCATA Challenge, which documents that forward trackers lose moving talkers or create ghost tracks during speech pauses. It also comes from the observation-guided adaptive particle filter, a forward tracker that falls back to a random-walk motion model when observations stop and so drifts across silences. The likelihood design comes from two sources. Multiple-Speaker Localization Based on Direct-Path Features discards time-frequency bins that fail a reciprocity test between microphone pairs. Reflection-Aware Sound Source Localization goes the other way and uses reflections as evidence, tracing rays backward off mapped walls. It reports that modeling reflected rays improves accuracy by 40% over direct rays alone. That paper also constrains us: its ray model assumes specular reflection above roughly 2 kHz, and it localizes single frames without any temporal reasoning about intent. We keep its core insight that a hypothesized position predicts where the echo peaks should appear. We apply it only to low-order image sources, which are mirror-image copies of the source across walls, and only in high-frequency bands, and we use it to score whole candidate trajectories rather than instantaneous fixes.
Research Program
The estimator has three parts.
The first is route enumeration. Given a 2D floor plan, we compute the distinct routes between small regions around the offset and onset estimates. A route counts as distinct if it passes a different set of obstacles on a different side. We prune any route whose shortest length is inconsistent with $\tau$ at plausible walking speeds.
The second is a per-route smoother. Each route gets a forward-backward particle smoother with a constant-velocity motion model constrained to that route's corridor. This makes each component unimodal in shape but still uncertain in timing.
The third is a reflection-consistent likelihood. For a candidate position, it predicts the direct-path and first-order image-source directions of arrival (the angles at which sound reaches the array) from the floor plan. It scores the observed spatial spectrum as a mixture of those peaks plus a diffuse floor. Because the fitted weight on the diffuse floor is higher in offset frames than onset frames, the asymmetric reliability of the two anchors falls out of the model rather than being hand-set. The route prior $p(h)$ starts uniform. In a second stage it is learned from simulated walking data conditioned on the anchor positions and headings. That learned prior is the only trained component.
The same mixture also forecasts. Run online during the gap, before speech resumes, it predicts which route the silent talker is on and where they will re-emerge. Comparing this online forecast with the retrospective version measures exactly how much the second anchor contributes.
Data come from two sources. The main testbed is SoundSpaces 2.0. It renders impulse responses on the fly by bidirectional path tracing, with diffraction and continuous crossfading for moving sources, so it avoids the popping artifacts of grid-sampled simulators. It also runs in Replica and Matterport3D scenes that contain real furniture and partitions, which create genuine route ambiguity. We script talkers who walk scene-graph routes with speech-and-silence patterns drawn from conversational statistics, recorded by a simulated circular array. Real-data checks use LOCATA's moving-talker tasks, with OptiTrack ground truth and natural speech pauses.
Baselines
The naive reference holds the last offset estimate constant through the gap, then jumps to the onset estimate. The second naive reference is straight-line interpolation between the two anchors. Multiagent off-screen behavior prediction in football tested linear and cubic splines and found them outperformed by learned imputers. Beating these is necessary but says little.
The forward-only acoustic baseline is our reimplementation of the observation-guided adaptive particle filter. It gets the same array geometry and spatial spectra, and its motion noise is tuned on the same validation scenes. The paper reports RMSE below 5° at 0 dB SNR on its simulated image-source room. That figure is for tracking during speech, not across silences. There is no reported result on Lacuna for silence-gap reconstruction on LOCATA or SoundSpaces, so every gap number here will be new, and all methods will be run by us under one protocol.
The strongest competitor is a map-constrained forward-backward particle smoother. It uses the same floor plan, the same motion model and the same number of particles as the proposal. It is run twice: once with a free-field likelihood that looks only at the direct path, and once with our reflection-consistent likelihood. It is strongest because it already uses both anchors and already allows multimodal posteriors in principle. The only thing it lacks is the explicit mixture over routes.
The strongest learned competitor is a Graph Imputer-style bidirectional recurrent network with a variational autoencoder. It is trained on the simulated SoundSpaces trajectories, takes per-frame spatial spectra as input, and gets the same training budget and hyperparameter search as the learned route prior. BRITS is trained the same way as the deterministic bidirectional reference.
Three ablations isolate the ingredients: removing the route mixture, which leaves a single smoother; replacing the reflection likelihood with direct-path bin masking in the style of the direct-path-features paper; and forcing equal anchor weights.
Beating the strongest competitor has a precise meaning. On scenes with at least two feasible routes, the proposal must place more posterior mass on the true route than the map-constrained smoother with the same likelihood, with a confidence interval computed over scenes. It must also achieve lower negative log-likelihood of the true gap trajectory. Lower average position error alone does not count.
Why not just…?
Why not just run a forward-backward particle smoother on the floor plan? This is the real threat. A smoother with enough particles represents route multimodality in principle. If it matches the proposal once both use the same likelihood, then the contribution reduces to the likelihood and the asymmetric-anchor finding, and we will report it that way. Our reason to expect a gap is practical. During a long silence, resampling tends to drive every surviving particle onto one route before the onset anchor arrives. The backward pass cannot then revive the route that died. An explicit route index keeps every feasible route alive until the evidence decides. The route-mass comparison at matched particle budgets, swept from small to large, settles this directly. We should see the gap shrink as the particle count grows, and we will report where it closes.
Why not just dereverberate first and track with a free-field model? A dereverberation front-end such as weighted prediction error estimates a fixed room filter. Maximum Negentropy Beamforming and the stationary-source analysis in Overcoming the Stationary Source Assumption describe how a fixed spatial filter paired with a moving source causes target cancellation. Where the direct path is blocked, Reflection-Aware Sound Source Localization shows that reflections are the only cue left. We will still run this pipeline, feeding weighted-prediction-error output into the free-field smoother. It is a cheap and fair test of whether explaining echoes beats removing them.
Why not just train an end-to-end generative imputer on simulated data? A Graph Imputer or a boundary-conditioned diffusion model could learn route choice and echo rejection implicitly, and with enough simulated rooms it might. The risk is transfer. It must generalize to unseen floor plans and to real arrays, and the VAST study found that simulator-trained localizers transfer only when the room acoustics were diverse in training. That study reports 4.5° real-world azimuth error for its reverberant-trained model against 12.2° for an anechoic-trained one. The held-out-scene split and the LOCATA transfer test below measure whether the learned imputer's advantage in familiar scenes survives in unfamiliar ones.
Proposed Experiments
The main SoundSpaces study stratifies test scenes by the number of feasible routes (one, two, or three or more), by gap length (one to ten seconds), and by reverberation level. Metrics are posterior mass on the true route, negative log-likelihood of the true path, best-of-six average displacement, and the fraction of sampled paths that cross walls. Scenes are split so that test floor plans never appear in training.
The first decisive check targets route choice. In one-route scenes, the proposal and the map-constrained smoother should tie. In multi-route scenes with gaps longer than a few seconds, the proposal should pull ahead. If it does not, the mixture adds nothing.
The second decisive check targets anchor asymmetry. Across reverberation levels, compare the localization error of offset and onset anchors against ground truth. Our hypothesis predicts that offset error grows with reverberation faster than onset error. If both grow equally, the physical argument is wrong and the asymmetric weighting should be dropped.
The third check targets the value of the future anchor. Compare the online route forecast during the gap with the retrospective posterior, reported as route-mass gain as a function of gap length.
Finally, on LOCATA moving-talker recordings, position error during natural pauses against OptiTrack ground truth tests whether the likelihood and the anchor weighting transfer to real arrays. For reference, LOCATA reports only about 1.0° azimuth error for the static single-source task, and no numbers for the moving tasks, which gives a sense of how much harder the moving case is. LOCATA rooms offer little route ambiguity, so this real-data test checks the acoustic components, not the mixture. We state that limitation openly.
References
- Multiagent off-screen behavior prediction in football
- Multiple Futures Prediction
- BRITS: Bidirectional Recurrent Imputation for Time Series
- MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction
- Uncertainty-Aware Driver Trajectory Prediction at Urban Intersections
- Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data
- HOME: Heatmap Output for future Motion Estimation
- BCDiff: Bidirectional Consistent Diffusion for Instantaneous Trajectory Prediction
- ProDiff: Prototype-Guided Diffusion for Minimal Information Trajectory Imputation
- AISFlow: Boundary-Informed Flow Matching for Long-Term AIS Trajectory Imputation
- MS-TIP: Imputation Aware Pedestrian Trajectory Prediction
- The LOCATA Challenge: Acoustic Source Localization and Tracking
- Multiple Concurrent Sound Source Tracking Based on Observation-Guided Adaptive Particle Filter
- Multiple-Speaker Localization Based on Direct-Path Features and Likelihood Maximization With Spatial Sparsity Regularization
- Reflection-Aware Sound Source Localization
- SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning
- Maximum Negentropy Beamforming
- VAST: The Virtual Acoustic Space Traveler Dataset