Physics-Checked Microphone Graphs for Synthetic-to-Real Speech Separation

When several phones and laptops in a room record the same conversation, a speech separation model has to decide how much to trust each device. Some devices are close to a talker. Others are muffled, clipped, or drifting in time. Current models make this decision with learned attention weights, which are scores computed from audio features. They learn those scores entirely from simulated rooms, where every microphone is perfectly synchronized, has a flat frequency response, and sits in an acoustically idealized space. We argue that this trust-weighting step is where synthetic-to-real transfer quietly breaks. A score learned from simulator features has never seen a real device's failure modes.

The proposal replaces learned trust with checked trust, and keeps everything else the same. Treat each microphone as a node in a graph. For each pair of microphones $i$ and $j$, ask a physical question that can be answered on any real recording without clean references: can the speakers separated at microphone $i$, each passed through a short linear room filter, reproduce what microphone $j$ actually recorded? How well they can becomes the edge weight. The same quantity also serves as a label-free training signal on unlabeled real recordings. In two sentences: the weights that decide which neighboring microphones to listen to should come from a cross-device physical consistency check rather than from features learned in simulation. That check is both a better trust signal under domain shift and a way to adapt the model on real data with no labels.

Core Claim

The two directions play different roles here. Dynamic neighbor weighting supplies the mechanism: aggregation that depends on the content of each neighbor rather than a fixed graph. The synthetic-to-real gap supplies the regime and the failure being targeted.

The gap itself is well documented. REAL-M collected crowdsourced real mixtures and showed that SepFormer drops from 20+ dB SI-SNR on synthetic benchmarks to about 2.8 dB on real mixtures, with 60.7% word error rate (WER) downstream. SI-SNR is the scale-invariant signal-to-noise ratio of the separated speech against a clean reference. WHAMR! showed that reverberation, not additive noise, is the dominant degradation. Low-Frequency Compensated Synthetic Impulse Responses showed that ray-traced room impulse responses are unrealistically flat below 1 kHz. These findings motivate the work.

The closest system is Neural Speech Separation Using Spatially Distributed Microphones. It interleaves dense self-attention across ad hoc microphone channels with per-channel recurrent layers, trains on 230,000 simulated impulse responses, and tests on a real meeting room with 4 phones and 3 laptops. It takes WER from 24.18% (multi-stream baseline) to 16.11%. Its channel attention is learned purely from simulated features, and it does nothing explicit about the domain gap.

Our method borrows its consistency signal from Reverberation as Supervision (RAS). RAS re-reverberates sources separated from the left channel through differentiable Wiener filters to predict the right channel. It uses that prediction only as a loss on a fixed stereo pair, assumes a time-invariant filter, and has to discard symmetric mixtures. We generalize the check to every edge of an arbitrary microphone graph, compute it over short time blocks, and use it as the aggregation weight itself, not only as a loss.

A theoretical result constrains the design. Graph Attention is Not Always Beneficial shows that feature-based attention does not beat plain averaging when node features are too noisy to separate good neighbors from bad ones. That is exactly the situation of simulator-trained features on real devices. It predicts that learned attention may collapse toward uniform averaging after transfer. A physically computed weight sidesteps this, because it does not rely on the shifted features to judge neighbors.

Research Program

The separator runs on each node with shared weights and produces $K$ source estimates $\hat s_i^{1..K}$ per microphone $i$, in the style of Wang et al. For each directed edge $(i,j)$ and each short block $b$ of roughly half a second, we fit per-source multi-tap linear filters $F^k_{ij}$ in closed form, as in RAS. The fit includes a small lag search, which absorbs clock offset and slow drift. A naive score fails in a predictable way: two microphones on the same laptop predict each other almost perfectly whatever the separation quality. We therefore normalize against a single-filter prediction from the raw mixture $x_i$. The edge score is how much the separation helps explain the neighbor:

$$ e_{ij}^{b} = \log \frac{\lVert x_j^b - \sum_{k} F^k_{ij} * \hat s_i^{k,b} \rVert^2}{\lVert x_j^b - G_{ij} * x_i^b \rVert^2}, $$

Here $x_j^b$ is microphone $j$'s recording in block $b$, $*$ denotes convolution, and $G_{ij}$ is the best single linear filter from the raw mixture at $i$. A more negative score means the separated sources explain microphone $j$ better than the raw mixture does.

Aggregation uses an exponential decay gate in the spirit of AdaptViG. That paper prunes weak patch connections with a smooth similarity-based gate rather than a hard threshold. We use a smooth gate on the physics score rather than on feature similarity. The weights are

$$ w_{ij}^{b} = \frac{\exp(-\max(0,\, e_{ij}^b + \delta)/\tau)}{\sum_{m \in \mathcal{N}_i} \exp(-\max(0,\, e_{im}^b + \delta)/\tau)}, $$

where $\tau$ is a learned temperature and $\delta$ a learned margin. These weights replace the dense attention in the inter-channel layers. A hybrid variant adds the physics term to the learned attention logits, so we can measure how much each source of trust contributes.

On unlabeled real recordings we adapt with the weighted reconstruction residual $\sum_{ij} w_{ij}\,\lVert x_j - \sum_k F^k_{ij} * \hat s_i^k\rVert^2$. To stop the model from routing all trust to one easy pair, this is regularized by an entropy floor on each node's weights. The program has three components: the physics-gated graph separator, block-wise filter fitting that tolerates movement and drift, and label-free adaptation on real multi-device audio.

Baselines

The naive reference is the same separator with uniform averaging over all microphones. This corresponds to an isotropic graph convolution like the one MESH2IR's encoder uses (MESH2IR). We include it because the CSBM analysis predicts that transferred learned attention may do no better than this. Beating it is the minimum bar.

The strongest existing method is the spatio-temporal dense channel-attention model of Wang et al., with MVDR beamforming. MVDR is a classical beamformer that preserves the target direction while minimizing output power. This model reports 16.11% WER on its real 7-device meeting room, against 24.18% for a multi-stream baseline. It is the strongest competitor because it already does dynamic neighbor weighting across ad hoc devices and was already tested sim-to-real.

We reimplement it with the same backbone depth, the same simulated training set, and the same simulator randomization (described below). We also give it the same unlabeled real audio, adapted with an unweighted pairwise RAS loss, so that any gain cannot be attributed to extra data. Hyperparameter search budgets are matched, in GPU hours and number of configurations.

Beating it means a lower WER and a higher reference-based SI-SDR on held-out real rooms, with the margin exceeding variation across random seeds. The advantage should grow, not shrink, as devices are degraded.

Three ablations isolate the key ingredient. Physics-only gating against learned-only attention against the hybrid tests where trust should come from. Removing the mixture-normalization denominator tests whether the co-located-microphone failure is real. Using the score only as a loss, not as an aggregation weight, separates "better training signal" from "better trust at inference."

Why not just…?

Why not just randomize the simulator harder? One could randomize per-device gain, frequency response, clock offset, and clipping, and apply the GMM-based low-frequency equalization of Tang et al., which cut far-field ASR WER from 15.56% to 14.19%. Then dense attention might learn robust trust on its own. This is the most serious threat, and we do not argue around it: every system in this proposal, ours included, is trained with this randomization. The remaining question is whether real failures fall outside the randomized family. Examples are a covered microphone, device-specific nonlinear processing, or low-frequency room modes that equalization only approximates in magnitude. If physics gating gives no gain over randomized dense attention on real rooms, the claim fails. We would report that outcome as such.

Why not just fine-tune the dense-attention model with a pairwise RAS loss on real data? This tests whether the consistency signal is useful only as a loss. It is exactly the "loss-only" ablation and the adapted version of the strongest baseline. RAS's own limitations motivate our changes. It needs a fixed stereo pair, assumes a time-invariant filter, and fails on symmetric mixtures, which its authors filtered out by hand. Our normalized, block-wise score is designed for those cases. Whether that matters on real ad hoc devices is an empirical question the ablation answers.

Why not just pick the best few microphones with a blind quality estimator and beamform? A blind SI-SNR estimator such as SSQP-Net (Lv et al., correlation about 0.9 with true SI-SNR) or the REAL-M CNN could rank channels. Lv et al. report that such estimators are unreliable at the very low SI-SNR typical of hard real mixtures. A hard top-$m$ selection also throws away spatial diversity, and Wang et al. found that performance grows with the number of devices. We run this baseline with the estimator scoring each channel's separated output, sweep $m$, and apply MVDR. If it matches us, the graph machinery is unnecessary.

Proposed Experiments

All systems train on simulated ad hoc rooms. Following WHAMR!, these are built with pyroomacoustics and LibriSpeech sources, use 2 to 8 randomly placed microphones, apply full device randomization, and use equalized impulse responses. A second training set uses MESH2IR-generated impulse responses to check that conclusions hold across simulators.

Real evaluation needs references. We therefore record in the LibriCSS style: known LibriSpeech utterances are played back through loudspeakers in several real rooms and captured by phones and laptops placed ad hoc, following Wang et al.'s protocol. Because the played-back dry signal is known, SI-SDR can be computed on real recordings, alongside ASR WER. We also report WER on LibriCSS, using channel subsets to vary array size. If the Wang et al. meeting-room data can be obtained, we report WER there as the direct comparison to their 16.11%. We found no reported absolute WER on Lacuna for LibriCSS with ad hoc subsets. The file gives only Lv et al.'s 2.8% relative improvement over Conv-TasNet at 40% overlap.

Three checks are decisive. The first is whether the edge score $e_{ij}$ on real recordings tracks true per-channel quality computed against the played-back reference better than learned attention weights do. A rank correlation comparison answers this, and it is the mechanistic test of the CSBM prediction. The second is a controlled physical-degradation test: in a real room we cover one device, move another to the far wall, and let a third drift in clock. Physics gating should down-weight these devices and lose little, while dense attention should degrade more. The third is the size of the sim-to-real gap, the difference between performance on matched simulated rooms and on real rooms, across all systems. The proposal succeeds if physics gating shrinks this gap relative to randomized dense attention and to estimator-based channel selection. It fails if randomization alone closes as much.