Calibration-Free RTF Denoising with Anchor-Tokenized Acoustic Graphs

A microphone array can pull one voice out of a noisy, echoing room, but only if it knows how that voice arrives at each microphone. That arrival pattern is the relative transfer function (RTF): for each frequency, the ratio between the sound at each microphone and the sound at a reference microphone. Below about 0 dB signal-to-noise ratio, standard RTF estimates degrade badly. peerRTF showed a fix. All RTFs in a given room lie on a low-dimensional manifold. If you build a nearest-neighbour graph over clean RTFs measured at thousands of positions and let a graph convolutional network pass messages between a noisy query and its clean neighbours, the result is a much better steering vector for the beamformer. At −10 dB this raised intelligibility (STOI) from about 24% unprocessed and about 66% for the standard estimator to about 72%. The catch is that the graph needs 4,104 pre-measured clean impulse responses from the same room (the MIRaGe grid). Nobody deploys a smart speaker by first scanning the living room.

We propose to remove that calibration step with an idea from scalable graph embedding. NodePiece shows that entities in a huge knowledge graph need not each own a learned vector. Each can be described by a few nearby "anchor" nodes and its distances to them, read by a small shared encoder. That makes the model inductive (it handles nodes never seen in training) and its size independent of the graph's size. We apply the same move to the acoustic manifold. The device passively collects a bank of RTF estimates from moments when it is confident a single talker dominates. It picks a small set of anchors from that bank. It then describes any new noisy RTF by the content of its nearest anchors, its distances to them, and their reliability. A room-agnostic encoder, trained across thousands of simulated rooms, maps that description to a denoised RTF, which feeds a mask-based MVDR beamformer: a linear spatial filter that passes the target direction undistorted and minimizes noise power. The claim is that most of peerRTF's low-SNR gain survives with no oracle grid. The anchor tokenization is what makes this work, not just a self-collected bank.

Core Claim

The relational structure that makes peerRTF work is "a noisy RTF is corrected by clean RTFs nearby on the room manifold." That structure can be learned once across rooms rather than memorized per room. It requires two things. The neighbours must be supplied as tokens with explicit relative-distance and reliability features rather than as fixed graph node identities. And the encoder must see only a fixed number $k$ of anchor tokens, so its parameters and per-query cost do not grow as the self-collected bank grows over hours of use.

We do not claim a memory saving. As Nimble GNN Embedding with Tensor-Train Decomposition documents, compression machinery on small graphs (ogbn-arxiv) is slower than a plain lookup table. An acoustic bank of a few thousand RTFs is small. The property we borrow from the scalability literature is inductiveness with a cost that does not depend on graph size. Compressing parameters is not the goal.

Research Program

Harvesting the bank. A mask network in the style of Improved MVDR Beamforming Using Single-Channel Mask Prediction Networks runs on each channel. A mask here is a per time-frequency weight in [0,1] marking speech versus noise. As in that paper, the masks are pooled across channels by element-wise maximum and given a floor. They weight speech and noise spatial covariance matrices, from which an RTF is estimated by generalized eigenvalue decomposition (GEVD) over short segments. A segment enters the bank only when the pooled speech mask is high and a simple dominance test passes, namely a large ratio between the first and second eigenvalues. Each bank entry stores the RTF and a scalar reliability, the estimated segment SNR.

Anchor tokenization. Anchors are chosen from the bank by farthest-point sampling under the Hermitian angle between RTFs, weighted toward high reliability. This parallels NodePiece's degree or PageRank anchor choice, but uses manifold coverage as the criterion. For a query RTF $\hat{\mathbf{d}}(f)$, the model receives its $k$ nearest anchors. Each anchor token is the anchor's RTF content, its angular distance to the query, and its reliability.

This differs from NodePiece in one respect that matters. NodePiece anchors are discrete IDs with learned embeddings, which would make the model room-specific again. Our anchors carry their physical content, so the encoder never memorizes identities. It is also why NodePiece's documented failure on sparse, large-diameter graphs (WN18RR), where far-away nodes get uninformative hashes, is a real risk for us. A talker standing far from every harvested position is the acoustic analogue, and we measure it directly.

Encoder and beamformer. A small set-transformer reads the query and its $k$ anchor tokens and outputs a residual correction to $\hat{\mathbf{d}}(f)$. Weights are shared across frequencies and microphone pairs, as in peerRTF. The denoised RTF and the mask-weighted noise covariance $\mathbf{R}_n(f)$ give the MVDR weights $\mathbf{w}(f)=\mathbf{R}_n^{-1}(f)\mathbf{d}(f)/(\mathbf{d}^H(f)\mathbf{R}_n^{-1}(f)\mathbf{d}(f))$. The beamformer stays linear, which avoids the waveform distortion that purely non-linear separators show. In Generalized Spatio-Temporal RNN Beamformer for Target Speech Separation, Conv-TasNet reached 22.07% WER against 11.36% for a beamforming approach.

Training. The encoder is trained on thousands of simulated rooms that vary in reverberation time, geometry, and source positions. Training banks are built by the same harvesting procedure at random SNRs, so the training conditions match deployment rather than oracle conditions. The loss is the Hermitian-angle error to the clean RTF, plus an SI-SDR term on the beamformed output. We add the SI-SDR term because MVDR-GCN gained PESQ while losing SI-SDR, and we want to see whether that trade-off appears here.

Baselines

The naive reference is the unprocessed reference microphone. Above it sits mask-based MVDR with a plain GEVD RTF, using exactly the harvesting masks described above. This is the "do nothing to the RTF" system. peerRTF reports it at about 66% STOI at −10 dB on MIRaGe, against about 24% unprocessed.

The strongest existing method is peerRTF with its full oracle grid. Its reported number is about 72% STOI at −10 dB on MIRaGe. It is the strongest competitor because it is the only cited method that denoises RTFs through manifold neighbours. However, it uses information our method forbids itself. We therefore treat it as a ceiling rather than a target to beat. Our success criterion is to exceed GEVD clearly and recover at least half of the 66%→72% gap with zero calibration measurements.

To make comparisons fair, we also run peerRTF in two degraded but deployable forms. The first uses a sparse oracle grid of 50 to 200 measured positions. The second is "blind peerRTF," in which its kNN graph is built over our harvested bank in place of the oracle grid. All learned systems share the mask network, the STFT settings, the MVDR back end, the test noises, and a matched hyperparameter search budget. For blind peerRTF, the per-room graph learning gets the same adaptation data our bank sees.

Ablations isolate the key ingredient. We replace farthest-point anchors with random anchors, drop the distance and reliability tokens, vary $k$, and train the encoder on a single room to test whether cross-room training rather than tokenization carries the gain. Beating the system means higher STOI and output SNR at −10 and −5 dB on held-out rooms, with SI-SDR no worse than GEVD-MVDR.

Why not just…?

Why not just build peerRTF's graph from the self-collected bank? This is the most serious threat, because it is our idea minus the tokenization. Our reason to expect it to fall short is that peerRTF's GCN is trained per room on clean neighbours. Harvested neighbours are noisy, and the number of them varies, and without reliability features the message passing cannot down-weight bad entries. Whether that matters is an empirical question. The blind-peerRTF baseline answers it directly. If it matches our model, the honest conclusion is that a calibration-free bank is the contribution and anchor tokenization is unnecessary, and we would report it that way.

Why not just train a neural RTF denoiser on the noisy RTF alone, across many rooms? A direct regressor learns a generic prior over RTFs. The manifold of any single room is much narrower than that prior, and peerRTF's gain over GEVD comes precisely from room-specific neighbours. The no-bank ablation (setting $k=0$) is this alternative. It tests whether anchors add room-specific information beyond a cross-room prior. We expect the gap to grow with reverberation time, where room manifolds diverge most.

Why not just improve the masks, for example with a stronger multichannel network or cross-channel attention? Better masks improve the covariance matrices, but at −10 dB the GEVD RTF is the bottleneck that peerRTF identified. Cross-channel attention as in Continuous Speech Separation with Ad Hoc Microphone Arrays addresses separation, not steering-vector priors. The two approaches are complementary. We check this by swapping in an oracle mask. If oracle masks alone close the gap to peerRTF, RTF denoising is the wrong lever.

Proposed Experiments

The primary benchmark is MIRaGe. Our model is trained only on simulated rooms and tested on MIRaGe positions, with its oracle grid used solely as ground truth for scoring. We sweep SNR from −10 to +10 dB and reverberation time across the 100 to 600 ms range, use NOISEX-92 noises unseen in training, and report STOI, DNSMOS, output SNR, PESQ, SI-SDR, and RTF angular error.

Secondary tests use simulated rooms from the ConferencingSpeech 2021 RIR pool held out from training, to probe unseen geometries. A stress test places talkers far from every harvested position, the analogue of NodePiece's large-diameter failure. We plot error against distance to the nearest anchor.

We also run a bank-growth experiment, feeding 1 to 60 minutes of harvested speech. If the claimed scaling property holds, quality should rise with bank size while per-query compute stays flat because $k$ is fixed. Blind peerRTF's graph cost grows with the bank.

The decisive checks are three. At −10 dB on MIRaGe, the method must clear GEVD-MVDR's ~66% STOI with zero calibration. It must beat blind peerRTF, or else we drop the tokenization claim. And its advantage over the no-bank denoiser must persist on held-out rooms. Failing the second check reduces the contribution to "harvest a bank," and failing the third reduces it to "a better learned prior." Both outcomes are informative, and the design separates them.