Hashing What Each Device Hears: Bit-Budgeted Mask Consensus for Ad-Hoc Microphone Arrays

When phones, laptops and smart speakers in a room are used together as one microphone array, a neural network can still clean up speech without knowing where the devices are. The network guesses, for every time-frequency cell of the recording, whether that cell is mostly speech or mostly noise. Those guesses, called masks, are then used to estimate how speech and noise spread across the devices, and a classical linear filter combines the channels. The practical bottleneck is the radio, not the arithmetic. Every current system of this kind assumes that devices exchange continuous floating-point audio or feature tensors, both to agree on the masks and to decide which devices are worth listening to. Over low-power wireless links shared by eight to sixteen devices, that is the expensive part.

We propose that each device send a short binary code per frame, around 64 bits, that summarizes what it hears. The codes are trained so that Hamming distance (the number of differing bits) preserves the neighbor relations of the device's continuous embedding, both across devices and across adjacent frames. A fusion node turns the set of codes into a shared mask and a ranking of devices. It then requests real audio only from the few devices the codes favor, and runs a mask-based multichannel Wiener filter on that subset. The claim is specific and testable. At a fixed bit budget, neighborhood-preserving codes produce masks that flicker less from frame to frame, leak less speech into the noise statistics, and keep the noise covariance better conditioned than plain sign quantization or uniform scalar quantization does. This holds especially when devices join, leave, or differ from those seen in training.

Core Claim and What Is New

Mask-based beamforming works without knowing the geometry because the spatial covariance matrices are accumulated directly from the masked observations: $\mathbf{R}_{nn}(f)=\sum_t M_n(t,f)\mathbf{X}(t,f)\mathbf{X}^H(t,f)/\sum_t M_n(t,f)$. Here $\mathbf{X}(t,f)$ is the vector of device signals at frame $t$ and frequency $f$, and $M_n$ is the noise mask. Improved MVDR Beamforming Using Single-Channel Mask Prediction Networks showed that a single consensus mask, taken as the element-wise maximum over channels, beats per-channel masks. On CHiME-3 real test data it lowered word error rate (WER) from 43.59% to 34.78%. The reason is that one leaked speech bin is enough to contaminate $\mathbf{R}_{nn}$. This observation motivates our design. Consensus needs information from every device, but only coarse information: which cells are speech, and which devices are clean.

The closest distributed system, DNN-Based Mask Estimation for Distributed Speech Enhancement in Spatially Unconstrained Microphone Arrays, has each node compress its microphones into one continuous audio stream. Nodes exchange these streams and refine masks for a multichannel Wiener filter. That gives about 3 dB of signal-to-interference improvement over single-node processing, and nearly 4 dB on the worst-placed nodes. It leaves two things open. The exchanged streams are float audio sent by every node regardless of quality, and the same stream does double duty for mask refinement and filtering. We separate those two roles. Bits carry the consensus; audio is sent only from the devices the bits select.

The binary-code machinery comes from hashing. K-Nearest Neighbors Hashing showed that minimizing per-coordinate quantization error flips bits for points near sign boundaries. Moving each point toward the centroid of its $k$ nearest neighbors before taking the sign cuts these flips, reaching 47.33% mAP at 16 bits on MNIST against 41.18% for ITQ. Local Semantic Structure Captured and Instance Discriminated by Unsupervised Hashing matches graph Laplacians between continuous and Hamming spaces. HCFRec pushes embeddings toward a two-peaked distribution so that the sign function rarely bites mid-mass. All of these were built for retrieval over static image or recommendation collections. What is new here is an acoustic reading of "bit flip near a boundary". It means a mask cell that toggles between adjacent frames. That toggle feeds noise into the recursively averaged covariance, $\mathbf{R}(t)=\alpha\mathbf{R}(t-1)+(1-\alpha)\mathbf{X}\mathbf{X}^H$ with forgetting factor $\alpha$, and produces musical-noise artifacts. Neighborhood preservation therefore has a concrete physical payoff that can be measured.

Research Program

Each device runs a small causal encoder on its own microphones. It produces a continuous embedding $\mathbf{e}_n(t)\in\mathbb{R}^D$ and transmits $\mathbf{b}_n(t)=\mathrm{sign}(\mathbf{e}_n(t))\in\{-1,1\}^B$. A 512-point STFT with a 16 ms hop gives 62.5 frames per second, so $B=64$ costs 4 kbps per device. For comparison, raw 16 kHz audio is 256 kbps and a one-bit-per-bin mask is about 16 kbps.

The fusion node runs a permutation-invariant set decoder over the codes. Because it pools over devices rather than querying a reference channel, it inherits channel-count invariance without the single reference-channel failure point of LABNet. It outputs a speech mask and noise mask on the time-frequency grid, plus a per-device reliability score, in the spirit of the stream attention in SDNet and the proximity scores in PickNet. The top $K$ devices by score stream their single-channel compressed audio, as in Furnon et al. A differentiable multichannel Wiener filter on those $K$ channels produces the output.

The whole system is trained end to end through the filter, following the differentiable-beamformer practice in Multi-talker Speech Separation Based on Permutation Invariant Training and Beamforming. The training objective is

$$\mathcal{L}=\mathcal{L}_{\text{SI-SNR}}(\hat{s},s)+\lambda\sum_{i}\sum_{j\in\mathcal{N}_k(i)}\big\|\tilde{\mathbf{e}}_i-\tilde{\mathbf{e}}_j\big\|^2 .$$

Here $\hat{s}$ and $s$ are the enhanced and clean signals, $\mathcal{L}_{\text{SI-SNR}}$ is the scale-invariant signal-to-noise loss, and $i$ indexes (device, frame) embeddings within a minibatch. $\mathcal{N}_k(i)$ is the set of $k$ nearest neighbors of $i$ in the continuous space, drawn from the same device's adjacent frames and from other devices in the same frame. $\tilde{\mathbf{e}}=\tanh(\beta\mathbf{e})$ is a relaxed code whose sharpness $\beta$ increases over training. The second term is a minibatch version of KNN shrinkage. It pulls neighborhoods together before the sign function so that they do not straddle boundaries. Restricting neighbors to the minibatch avoids the quadratic offline graph construction that the hashing literature lists as its main cost.

The analysis side tests the mechanism directly. We measure the frame-to-frame mask toggle rate, the fraction of true speech energy assigned to $M_n$, and the condition number of $\mathbf{R}_{nn}(f)$ over time. All three are compared across code types at matched bits. If neighborhood preservation lowers word error rate but not these three quantities, the proposed mechanism is wrong and we will say so.

Baselines

The naive reference point is local-only processing. Each device estimates its own mask and filters its own microphones, as in the single-node DNN stage of Furnon et al. This costs zero bits. It sets the floor that any exchange scheme must clear.

The strongest distributed competitor is Furnon et al.'s full two-step scheme, in which every node streams continuous compressed audio and refines masks from it. We will re-run it on the same Pyroomacoustics configurations, same LibriSpeech speech, same noise, same node counts, and the same training compute and tuning budget as our system. It has the same downstream filter and differs only in what is exchanged. Its reported gain is about 3 dB average SIR improvement over single-node processing. Our target is to land within 0.5 dB of it, under our re-run, while transmitting under one tenth of its bits. Beating it on absolute quality is not the goal.

The strongest centralized reference is LABNet with every channel's audio. It reports PESQ 2.92 on 6-microphone WSJ0+WHAM!, with 52k parameters and 0.316 GMACs. We will train it on our data at matched compute and treat it as an upper bound that ignores bandwidth. The result we care about is the PESQ-versus-bits curve. On the same benchmark, SDNet's 16.11 dB SI-SNRi on 6-microphone two-speaker LibriSpeech separation is the reference for the overlapping-speech condition.

The ablation that isolates the key ingredient keeps our architecture and bit budget and removes the neighborhood term ($\lambda=0$). The codes are then learned purely from the task loss through a straight-through sign. A second matched-budget ablation replaces sign codes with uniform 4-bit scalar quantization of a 16-dimensional embedding, which mimics quantization-aware training. We count beating them as a significant reduction in mask toggle rate and $\mathbf{R}_{nn}$ condition number, plus better WER or PESQ in at least the churn and unseen-device conditions.

Why not just…?

Why not just train the binary bottleneck end to end with a straight-through estimator and drop the neighborhood loss? This is the most serious threat. The task loss already says which distinctions matter, and on in-distribution data it may match our method. Our bet is that it fails where the task loss is silent. One such case is frame-to-frame stability: the filter's temporal averaging hides flicker from the loss while it still adds variance to the covariance. The other is devices or channel counts unseen in training, where nothing anchors a new device's codes to the familiar geometry. KNN Hashing's evidence that boundary flips, not average error, drive degradation supports the bet, but only for images. The $\lambda=0$ ablation under churn and unseen hardware will settle it. If the two methods tie there, we will report that bits suffice and that neighborhood preservation does not.

Why not just exchange a binarized mask and take the element-wise maximum, as Erdogan et al. do? Max-consensus is principled and has strong CHiME-3 evidence. It costs roughly four times our budget per device at the frame rate above, and it carries no information about device quality, so it cannot decide whose audio to request. We include it at its native bitrate and at a frequency-downsampled bitrate matched to ours. If it matches our codes at matched bits, the learned codes are unnecessary.

Why not just pick the best device with PickNet and skip beamforming? PickNet lowers WER from 16.4% to 12.6% on real handheld recordings at 1.32 ms per frame. It assumes one active talker, though, and gives up spatial nulling. Libri-adhoc40 reports a 46.6% relative WER reduction going from 10 to 40 channels, which suggests that combining channels still pays. Our $K=1$ setting is essentially code-driven selection. The $K$ sweep on overlapping and single-talker subsets will show where selection alone is enough. We expect it to be enough on single-talker segments, and we will report that honestly.

Why not just INT8-quantize the exchanged audio or features? Eight bits per value keeps the full dimensionality and buys only fourfold compression. The 4-bit scalar ablation tests the stronger version of this alternative at matched bits.

Proposed Experiments

The main testbed reproduces the Furnon et al. simulated multi-node setup with node counts from 4 to 16. We add two stress conditions. In the churn condition, devices drop out or appear mid-utterance. In the unseen-hardware condition, test devices have microphone responses and gains not seen in training. WSJ0+WHAM! with 2 to 12 microphones allows direct comparison with LABNet. The LibriSpeech two-speaker ad-hoc separation task from SDNet tests overlapping speech. Real recordings from Libri-adhoc40, which are synchronized, give WER with a pretrained recognizer. Clock drift is deliberately left out of scope so that it does not confound the code comparison.

Metrics cover four areas. Quality is measured by SI-SNR improvement, SIR, PESQ and WER. Bandwidth is bits per second per device, counting both codes and requested audio. Edge cost is encoder parameters and MACs, kept at or below LABNet's scale. Mechanism is measured by toggle rate, speech leakage into $M_n$, and the median and 95th-percentile condition number of $\mathbf{R}_{nn}$.

The decisive checks are three. First, quality must follow a curve over $B\in\{16,32,64,128\}$ and $K\in\{1,2,3,N\}$ that reaches the full-streaming Furnon baseline at a small fraction of its bits. Second, the neighborhood term must beat the $\lambda=0$ and scalar-quantization ablations on the mechanism metrics, and the gap must widen under churn and unseen hardware. Third, on Libri-adhoc40, code-driven selection of $K$ devices must beat both PickNet-style $K=1$ and Erdogan-style max-consensus at matched bits. If the first check holds and the second fails, the paper narrows to "short binary codes are enough for distributed mask consensus", which is still a useful finding.