Phone calls and earbud calls often carry speech that is both narrowband, cut to roughly 300 Hz–3.4 kHz at an 8 kHz sampling rate, and noisy. Neural "bandwidth extension" models, which invent the missing high frequencies from the narrowband signal, work well on clean input. In loud places they fail in a characteristic way: they remove noise by also removing quiet speech. Many earbuds and glasses already carry a second sensor, a bone-conduction microphone or accelerometer. It picks up skull vibration, so it ignores airborne noise, but it only hears frequencies below about 800 Hz–3 kHz. This note borrows an idea from cosmology to tie the two sensors together. Cosmologists multiply maps from two instruments whose noise has unrelated physical causes. In that product the noise averages to zero and only the shared signal survives. Applied to speech, the averaged cross-spectrum between the air microphone and the vibration sensor, over the frequencies both hear, gives a noise-free estimate of how much speech energy the air channel really contains. It also gives each wearer's own sensor transfer function, and it does both without a clean reference recording.
The proposal uses that cross-spectrum for three jobs. It conditions a bandwidth-extension generator. It serves as a training loss that needs no labels and penalizes over-suppression. It serves as an evaluation metric for real recordings that have no clean reference. The central claim is specific and can be shown false. Explicit, per-utterance cross-spectral calibration makes air/bone fusion robust to mismatch between the bone-conduction transfer function assumed in training and the one on the actual wearer. Networks that fuse the two raw sensor streams end to end do not have that robustness.
Why the cosmology analogy is load-bearing
In large-scale-structure surveys, a galaxy map's clustering amplitude is a product of galaxy "bias" and the true matter fluctuation amplitude, and one map alone cannot separate the two. SDSS galaxy bias from halo mass-bias relation and its cosmological implications and Giusarma et al. (2018) break this degeneracy by correlating the biased tracer with a lensing probe, which responds to mass directly. Pullen et al. (2016) builds a ratio statistic, $E_G$, in which the unknown bias cancels. Joint analysis of DES Y3 and CMB lensing from SPT and Planck II makes the noise argument explicit. Optical and microwave systematics share no physical origin, so they vanish in the cross-correlation, which there reached a detection signal-to-noise of 23.9.
These papers supply the method, an estimator design. They do not supply a model. The same work also sets a constraint. In the DES×CMB-lensing analysis, small scales had to be cut because the model failed there. The speech counterpart is that cross-spectra are only meaningful in the band both sensors hear, and only after enough averaging.
Write the air channel as $Y = G X + N_a$ and the vibration channel as $B = H X + N_b$. Here $X$ is clean speech in one frequency bin, $G$ and $H$ are the sensors' transfer functions, and $N_a, N_b$ are noises with no shared cause. Then
$$\mathbb{E}[Y B^*] = G H^* S_x, \qquad \mathbb{E}[|Y|^2] = |G|^2 S_x + S_{n_a},$$
where $S_x$ is the clean-speech power and $S_{n_a}$ the air-noise power. The single-channel quantity on the right mixes speech and noise inseparably. This is the speech version of the bias–amplitude degeneracy, and it is why single-microphone denoisers at low signal-to-noise ratio (SNR) cannot tell quiet formants from noise. The cross term carries no air noise.
Combining the cross term with the bone auto-spectrum gives two estimates. The first is $|G|^2 S_x \approx |\mathbb{E}[YB^*]|^2 / (\mathbb{E}[|B|^2]-S_{n_b})$, a noise-unbiased estimate of speech power in the air channel. The second is $G/H \approx \mathbb{E}[YB^*]/\mathbb{E}[|B|^2]$, the relative transfer function for this wearer. The variance of these estimates falls roughly as $1/K$ in the number $K$ of averaged frames, because averaging independent frames shrinks variance that way. That sets the main design tension. Speech changes quickly, so windows must stay short, while the estimate needs many frames.
What prior speech work does and does not do
Air/bone fusion itself is established, so this proposal does not claim it. Bone-Conduction Sensor Assisted Noise Estimation uses the bone sensor as a binary gate on speech-presence probability inside classical noise trackers, improving SRMRnorm in a café from 2.64 to 3.72. It never models cross-spectra and never extends bandwidth. A graphical model for multi-sensory speech processing is probabilistically principled. It assumes frequency bins are independent and reconstructs nothing above the bone sensor's cutoff. Time-Domain Multi-Modal Bone/Air Conducted Speech Enhancement and VibOmni fuse raw streams in deep networks. VibOmni trains on synthetic vibration generated from an idealized Gaussian bone-conduction function, and the file itself flags that idealization as a limitation. Neither paper handles a narrowband air channel. These works motivate the proposal, because they show the second sensor helps. They also define its main competitor, implicit learned fusion.
On the bandwidth-extension side, the method builds most directly on A Neural Codec Approach for Noise-Robust Bandwidth Expansion. That model predicts tokens of a neural audio codec (a learned compressor whose decoder turns discrete tokens back into audio) from noisy 8 kHz speech, and it is documented to over-suppress low-energy formants at low SNR. That failure is exactly the degeneracy written above. Towards High-Quality and Efficient Speech Bandwidth Extension With Parallel Amplitude and Phase Prediction (AP-BWE) and A Two-Stage Approach to Speech Bandwidth Extension are strong single-channel generators that assume clean input. French Listening Tests for Body-Conducted Speech Enhancement motivates the identity angle. Single-sensor body-conducted enhancement (EBEN) cost female throat-microphone speakers a median 11% in identification. The same paper shows ECAPA2 similarity tracks human identity judgments (ρ = 0.90), which lets us measure identity objectively.
Research Program
The setting is an earable or headset whose air microphone path is narrowband (8 kHz, as on telephony and Bluetooth hands-free links) and noisy, paired with a vibration sensor below 800 Hz in the VibOmni configuration, or up to roughly 1–3 kHz for in-ear or throat sensors. The target output is denoised 16 kHz speech.
We will build three components. The first is a streaming cross-spectral front end. It recursively averages the air auto-spectrum, the bone auto-spectrum and the air–bone cross-spectrum over the overlap band. From these it produces coherence, a noise-unbiased speech-power estimate, and the relative transfer function, all with a tunable averaging time constant.
The second component injects these quantities as conditioning features into the codec token predictor of Liu et al. The predictor already receives the noisy narrowband signal and raw vibration. The point is to anchor how much speech energy the predictor keeps where the air channel is ambiguous.
The third component is a cross-spectral consistency loss. The generated output, band-passed to the overlap region, should reproduce the measured air–bone cross-spectrum after correction by the estimated transfer function. Because air noise is absent from that target in expectation, the loss requires no clean reference. Over-suppressing formants lowers the output's cross-power with the bone channel, so the loss penalizes exactly the documented failure. The same quantity, reported as a ratio of output cross-power to measured cross-power, becomes a reference-free over-suppression score for in-the-wild recordings.
Training uses LibriSpeech speech, DNS-Challenge noise, simulated narrowband channels, and synthetic vibration from Gaussian bone-conduction functions in the style of VibOmni. The consistency loss also allows a second stage of label-free fine-tuning on real paired recordings, where no clean target exists.
Baselines
The naive reference point is the noisy narrowband signal upsampled to 16 kHz with no processing. It fixes the floor for every metric and checks that the evaluation pipeline is correct.
The strongest single-channel competitor is the codec approach of Liu et al., retrained on our training mix with its two-stage schedule. It is the strongest because it is the most recent method built specifically for noisy narrowband input, and it is the model our generator extends, so any gain is directly attributable. No reported result found on Lacuna for its DNSMOS values; the file says only "competitive." AP-BWE and the two-stage model of Lin et al. will be run as clean-input generators, both with and without a front-end denoiser. For Lin et al. the file reports WB-PESQ of 4.08 on clean Valentini-Botinhao bandwidth extension and LibriSpeech word error rate (WER) of 12.67%, against an 11.19% clean oracle. Those clean-input figures give a ceiling, not a noisy-condition target.
The strongest fusion competitor is a dual-stream network of the VibOmni type, which fuses raw air and vibration inputs, with the same codec decoder head. It gets the same parameter count, the same data and the same hyperparameter search budget as our model. Two versions will be trained, one with VibOmni's Gaussian bone-conduction augmentation and one with a much more widely randomized family of bone-conduction functions. VibOmni reports up to a 21% PESQ gain over FullSubNet and about 40% lower WER in the wild. There is no absolute figure in the file, so we reimplement it rather than copy numbers. The late-fusion network of Yu et al. (PESQ 1.735 versus 1.594 air-only on TMHINT) and the binary-gating scheme of Lee et al. serve as lighter fusion references.
The key ablations hold the network fixed and vary one thing at a time. We remove the cross-spectral features, remove the consistency loss, replace the estimated transfer function with an oracle one, and replace cross-spectra with bone auto-spectra alone. The last of these isolates whether the cross term, rather than simply more bone information, does the work.
Beating the baselines means two things. Under matched bone-conduction conditions, our model should be no worse than raw fusion. Under mismatched conditions, it should do significantly better on PESQ, ECAPA similarity and WER, with a smaller drop from matched to mismatched than raw fusion shows.
Why not just…?
Why not just train an end-to-end dual-input network and let it learn the cross-correlation implicitly? This is the real threat. A convolutional network given both streams can in principle compute cross-spectra. Our bet is that it learns cross-spectra under the training transfer functions. VibOmni's own reliance on an idealized Gaussian bone-conduction function, and the per-sensor variability in the Vibravox listening tests, suggest real wearers fall outside that family. The mismatch experiment below settles it. If the widely randomized raw-fusion baseline matches our mismatch robustness, the explicit front end adds nothing. In that case the contribution shrinks to the label-free loss and metric, and we would report that.
Why not just use a classical coherence-based dual-channel Wiener filter? It would clean the overlap band, but it cannot create content the sensors never captured. That includes everything above the vibration sensor's cutoff in the air channel and everything above 4 kHz in narrowband speech, where fricatives like /s/ and /f/ live. It is still a useful component baseline. We will run it as a front end feeding AP-BWE, which tests whether the gain comes from the cross-spectral idea or from coupling it to a generator.
Why not just train the single-microphone codec model on more noise data? Equation 1 says the low-SNR ambiguity is structural. With one channel, quiet speech and noise produce the same auto-spectrum. More data sharpens the prior but cannot resolve the ambiguity, which is consistent with the over-suppression Liu et al. report despite two-stage training. We will test this directly by doubling the single-channel training data and checking whether the formant-suppression gap at −5 to 0 dB closes.
Why not just use the bone sensor as a speech-presence gate, as Lee et al. do? A gate says whether speech is present, not how much. Over-suppression is a problem of amount, so the gate-based baseline should help noise tracking but not formant retention. Our ablations measure this directly.
Proposed Experiments
The decisive experiment is transfer-function mismatch. Train all fusion models on synthetic Gaussian bone-conduction data, then test on real paired recordings from Vibravox: the forehead accelerometer, the rigid in-ear microphone and the throat microphone, with the airborne headset as reference. The air channel is narrowbanded and mixed with DNS noise at −5 to 10 dB. We report WB-PESQ, STOI, DNSMOS, log-spectral distance, ECAPA speaker similarity broken out by speaker sex, and ASR WER. The prediction is that raw fusion degrades markedly from synthetic to real sensors while cross-spectral anchoring degrades less, most visibly on female throat-microphone identity, where EBEN already failed.
The second experiment validates the metric. On held-out simulated data, where clean references exist, we check whether the reference-free cross-power ratio predicts PESQ and ECAPA drops across all systems. If it does, we apply it to unlabeled in-the-wild recordings.
The third experiment tests label-free adaptation. We fine-tune with the consistency loss alone on real Vibravox pairs, with no clean targets used, and measure the gain on held-out speakers.
The fourth experiment checks the averaging time-constant trade-off implied by the $1/K$ variance argument. We sweep the window, report quality against algorithmic latency, and verify that the front end fits a streaming budget comparable to Streaming SEANet's sub-18 ms latency from Real-time Speech Frequency Bandwidth Extension.
The proposal fails cleanly in two cases. One is that randomized raw fusion closes the mismatch gap. The other is that the cross-power ratio does not track reference-based quality.
References
- SDSS galaxy bias from halo mass-bias relation and its cosmological implications
- Scale-dependent galaxy bias, CMB lensing-galaxy cross-correlation, and neutrino masses
- Constraining gravity at the largest scales through CMB lensing and galaxy velocities
- Joint analysis of Dark Energy Survey Year 3 data and CMB lensing from SPT and<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:mi>P</mml:mi><mml:mi>l</mml:mi><mml:mi>a</mml:mi><mml:mi>n</mml:mi><mml:mi>c</mml:mi><mml:mi>k</mml:mi></mml:mrow></mml:math>. II. Cross-correlation measurements and cosmological constraints
- Bone-Conduction Sensor Assisted Noise Estimation for Improved Speech Enhancement
- A graphical model for multi-sensory speech processing in air-and-bone conductive microphones
- Time-Domain Multi-modal Bone/air Conducted Speech Enhancement
- VibOmni: Towards Scalable Bone-conduction Speech Enhancement on Earables
- A Neural Codec Approach for Noise-Robust Bandwidth Expansion
- Towards High-Quality and Efficient Speech Bandwidth Extension With Parallel Amplitude and Phase Prediction
- A Two-Stage Approach to Speech Bandwidth Extension
- French Listening Tests for the Assessment of Intelligibility, Quality, and Identity of Body-Conducted Speech Enhancement
- Real-Time Speech Frequency Bandwidth Extension