Turning the Room Off: Separating Mumbling from Reverberation in Spontaneous Speech

Conversational speech is hard for recognizers for two different reasons that leave similar marks on the audio. Speakers reduce their articulation: they talk fast, undershoot vowel targets, and drop consonants. Rooms also smear the signal through reverberation and noise. Both effects blur spectral detail over time, so a recording that sounds "mushy" gives no direct indication of whether the speaker mumbled or the room echoed. This matters for diagnosis and for design. You cannot know whether to spend effort on pronunciation modeling or on channel robustness if you cannot tell which effect is costing you errors. Heavy-ion physics has a mature recipe for this kind of confound, and we propose to borrow its design logic, not its physics. That recipe compares against a system where the "medium" is switched off, uses a probe the medium cannot touch, and then tests whether the observed suppression factorizes into the two contributions.

The speech version is unusually clean, because here the medium can be switched on and off at will. Electromagnetic articulography, or EMA, uses small coils glued to the tongue, lips, and jaw and tracked by magnetic induction. It records how the mouth actually moved, and that measurement is untouched by room acoustics. Reverberation can be added to the matching audio by convolving it with a room impulse response, so every utterance has an exact counterpart with identical true kinematics but a degraded channel. We use this to build and calibrate an acoustic-to-articulatory inverter, a model that predicts articulator trajectories from audio. The target property is that the inverter's estimate of articulatory reduction stays the same when reverberation is added. We then use it as a measuring instrument on spontaneous speech that has no EMA. The new element is the calibration protocol and the training signal it implies. On unlabeled spontaneous audio, we cannot know the true kinematics, but we do know they must not change when we add a room.

Core Claim

An inverter can be made into a channel-invariant reduction meter, and the claim is testable in a factorial design borrowed from proton–nucleus physics. PHENIX separated cold-nucleus effects from hot-medium effects in two steps. First, it ran p+Al and p+Au collisions where no hot medium forms. It found that away-side dihadron widths broaden in p+Au with the number of binary collisions, while near-side widths are unchanged (Nonperturbative-transverse-momentum broadening in dihadron angular correlations, Nonperturbative-transverse-momentum broadening in dihadron angular correlations in s N N = 200 GeV proton-nucleus collisions; Nonperturbative-transverse-momentum broadening in dihadron angular correlations in s N N = 200 GeV proton-nucleus collisions). Second, it used direct photons, which ignore the colored medium, as an unmodified reference. Photon-triggered correlations in p+Au and d+Au show no change in yield or width (PHENIX results on jet modification with π0- and photon-triggered correlations, PHENIX results on jet modification with π0- and photon-triggered two particle correlations in p+p, p(d)+Au, and Au+Au collisions; PHENIX results on jet modification with π0- and photon-triggered two particle correlations in p+p, p(d)+Au, and Au+Au collisions). These papers motivate the design; they supply no technique we reuse.

In our setting, the "intrinsic" factor is speaking rate, which induces reduction. The "medium" is the reverberant channel, and the EMA trace plays the photon's role. Let $\rho(r,c)$ be inversion correlation at speaking rate $r$ (normal or fast) and channel $c$ (clean or reverberant). We define an interaction term

$$I = \log\frac{\rho(\text{fast},\text{rev})}{\rho(\text{normal},\text{clean})} - \log\frac{\rho(\text{fast},\text{clean})}{\rho(\text{normal},\text{clean})} - \log\frac{\rho(\text{normal},\text{rev})}{\rho(\text{normal},\text{clean})},$$

which is zero when the two degradations act independently. A nonzero $I$ is analogous to the finding that suppression in large systems is not just cold effects times hot effects.

We also define a recovered reduction factor $K = A_{\text{fast}}/A_{\text{normal}}$, where $A$ is the displacement range of a tract variable such as tongue-tip constriction. This is the speech counterpart of $R_{pA}$, the nuclear modification factor that compares proton–nucleus to proton–proton yields. Because EMA gives the true $K$, an instrument is calibrated when its estimate $\hat K$ matches the true $K$ under every channel. We make two predictions. Current inverters will show $I<0$, meaning reverberation hurts most on already-reduced speech, and their $\hat K$ will drift toward "more reduced" under reverberation, because reverberation masquerades as mumbling. A paired channel-consistency objective, applied on unlabeled spontaneous audio, should remove most of that drift.

Research Program

The labeled foundation is the EMA corpora that vary speaking rate. HPRC provides eight speakers across production rates, and EMA-IEEE provides eight speakers at normal and fast rates. MOCHA-TIMIT is pooled with HPRC to match the 16-speaker setting of Speaker-Independent Acoustic-to-Articulatory Speech Inversion. That paper combines HuBERT features (a self-supervised speech encoder), speaker embeddings, and adversarial training to reach speaker-independent inversion on read speech. We reuse its architecture family as our starting inverter. We add simulated room impulse responses spanning a range of reverberation times, applied to held-out speakers, to form the four factorial cells. EMA ground truth is identical across the channel axis by construction.

The instrument is trained with three terms. The first is the usual regression to EMA trajectories on clean and reverberant copies of labeled read speech; this is multi-condition training. The second is a consistency term asking that predicted trajectories for an utterance and its reverberated copy agree. The third is the key departure: that same consistency term applied to unlabeled spontaneous audio from the Buckeye corpus (The buckeye corpus of speech). This exposes the model to genuine conversational reduction, which no EMA corpus contains, while supervising only what we are sure of. Room acoustics do not move the tongue.

The calibrated instrument is then applied to Buckeye. Its phonetic labels mark surface reductions such as deletions and vowel centralization relative to canonical forms, which provides an external check that $\hat K$ tracks real reduction. It is also applied to Switchboard, where we attribute per-utterance recognizer errors to estimated reduction versus estimated channel severity.

Hyperparameter optimization acts as a constraint on the science, not as a source of gains. Existing inverters use fixed hand-chosen topologies, such as five-layer, 1024-unit networks in Multi-Corpus Acoustic-to-Articulatory Speech Inversion and Noise Robust Acoustic to Articulatory Speech Inversion. Quantifying the Hyperparameter Sensitivity of Neural Networks for Character-level Sequence-to-Sequence Tasks showed that architecture rankings flip once search budgets are equalized, and MolRep showed that rankings across 16 architectures depend on tuning. A claim such as "$I<0$ for multi-condition training but $I\approx 0$ for our objective" could therefore be a tuning artifact. Every compared system gets the same tree-structured Bayesian search budget, and hyperparameters are selected on held-out speakers, never on the factorial test cells, to avoid the validation overfitting documented in Overtuning in Hyperparameter Optimization. We also report, alongside the best configuration, the fraction of near-optimal configurations for which each scientific conclusion holds. This uses the "percentage of runs within an equivalence margin" idea from Wiemerslage et al. as a robustness check on conclusions.

Baselines

The do-nothing reference is a clean-trained inverter applied to reverberant and spontaneous audio. On noisy XRMB, the noise-robust inversion paper reports that a clean-trained model falls from 0.782 to 0.640 average correlation. That paper used additive noise, not reverberation. For reverberant speech, we found no reported result on Lacuna for any inverter. For the factorial interaction term, the reference value is zero, meaning independent degradations. For $\hat K$, the naive reference is the uncorrected inverter's estimate.

The strongest existing method on the read-speech axis is the HuBERT-based speaker-independent inverter of Sitaram et al. It reports a Pearson correlation of 0.784 on the combined MOCHA-TIMIT plus HPRC dataset (Speaker-Independent Acoustic-to-Articulatory Speech Inversion). We first reproduce it on clean data within our tuning budget and require reaching roughly that figure before trusting any comparison.

The strongest competitor on the channel axis is multi-condition training, following Noise Robust Acoustic to Articulatory Speech Inversion. That paper reports 0.70949 correlation on noisy XRMB but 0.741 on clean speech, down from 0.782. We rebuild multi-condition training with reverberation on the HuBERT inverter, with the same data, impulse responses, and search budget as our method. This is the competitor most likely to tie us.

Two ablations isolate the key ingredient. The first applies the consistency term only on labeled read speech, which tests whether paired invariance helps beyond plain augmentation. The second applies it on Buckeye as well, which tests whether exposure to real spontaneous reduction matters. Beating the multi-condition baseline requires three outcomes. Clean correlation must not fall below the baseline. $|\hat K - K|$ under reverberation must be reduced on held-out EMA speakers, with confidence intervals across the near-optimal configuration set. Agreement between $\hat K$ and Buckeye-annotated reductions must survive added reverberation where the baseline's agreement degrades.

Why not just…?

Why not just dereverberate the audio first? The only evidence in our file argues against it. The noise-robust inversion study found that log-MMSE enhancement before a clean-trained inverter made correlation worse, consistent with enhancement erasing the fine temporal cues inversion needs. That result concerns noise and a classical enhancer, not modern dereverberation, so it does not settle the question. We include a dereverberation front-end plus clean inverter as an additional baseline in the factorial. If it yields $I\approx0$ and a stable $\hat K$, the consistency objective is unnecessary and we will say so.

Why not just use multi-condition training, which already shares ground truth across channels? This is the real threat. Multi-condition training already sees paired clean and reverberant examples with the same targets, and it may already be channel-invariant enough. Our extra ingredient has bite only where labels do not exist, which is spontaneous speech. The read-only versus Buckeye ablation is designed to show whether that matters. If multi-condition training matches us on $\hat K$ stability and Buckeye agreement, the contribution narrows to the calibration protocol and the attribution analysis. That is still a usable result, but a smaller one.

Why not skip articulation and measure reduction directly from self-supervised features or recognizer confidence? HuBERT features are already the inverter's input, so a feature-space distance from read speech is a cheaper reduction proxy. The concern is that such features encode the channel as well as the speaker, which is exactly the confound at issue. We test this rather than assume it. The same factorial and Buckeye checks are run on a feature-distance measure. If that measure is equally stable under reverberation and equally predictive of annotated reductions, the articulatory bottleneck adds nothing and we report that.

Why not treat the nuclear analogy as decoration? We use only three things from it: a cell where one effect is absent, a probe immune to the medium, and a factorization test. Each maps onto a concrete construction above, and none depends on any physics claim. The non-linearity seen in nuclei, with no broadening in p+Al but measurable broadening in p+Au, motivates looking for a nonzero interaction term rather than assuming additivity.

Proposed Experiments

The first experiment is the factorial on held-out HPRC and EMA-IEEE speakers. For each system, we compute $\rho$ in all four cells across several reverberation times, then $I$ and $\hat K$ per tract variable. Confidence intervals come from bootstrapping over speakers and from the near-optimal configuration set. The decisive check is whether the reverberation-induced drift of $\hat K$ toward "more reduced", which we predict for clean and multi-condition inverters, shrinks under the Buckeye consistency objective without loss of clean correlation relative to the reproduced 0.784 setting.

The second experiment validates on genuine spontaneous speech. On Buckeye, we correlate utterance-level $\hat K$ with the rate of annotated surface reductions, then repeat after adding synthetic reverberation. A good instrument keeps the correlation, while a confounded one loses it or inflates its reduction estimates. No reported result was found on Lacuna for inversion on Buckeye, so these are first measurements.

The third experiment is attribution on Switchboard. We regress per-utterance word errors of a fixed recognizer on estimated reduction and estimated channel severity. We use the corrected references described in Toward Zero Oracle Word Error Rate on the Switchboard Benchmark, because that paper shows reference errors alone moved one system from 4.3% to 2.3% WER, and uncorrected references would contaminate any attribution. Telephone bandwidth differs from EMA audio, so this experiment is exploratory and reported as such.

The project fails cleanly in any of three cases. Multi-condition training or a dereverberation front-end matches the consistency objective on every check. Feature distance matches the articulatory meter. Or no system's $\hat K$ tracks Buckeye reductions even on clean audio.

References

  1. Nonperturbative-transverse-momentum broadening in dihadron angular correlations in <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:msqrt><mml:msub><mml:mi>s</mml:mi><mml:msub><mml:mrow/><mml:mrow><mml:mi>N</mml:mi><mml:mi>N</mml:mi></mml:mrow></mml:msub></mml:msub></mml:msqrt><mml:mo>=</mml:mo><mml:mn>200</mml:mn><mml:mspace width="0.16em"/><mml:mi>GeV</mml:mi></mml:mrow></mml:math> proton-nucleus collisions
  2. Nonperturbative-transverse-momentum broadening in dihadron angular correlations in <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:msqrt><mml:msub><mml:mi>s</mml:mi><mml:msub><mml:mrow/><mml:mrow><mml:mi>N</mml:mi><mml:mi>N</mml:mi></mml:mrow></mml:msub></mml:msub></mml:msqrt><mml:mo>=</mml:mo><mml:mn>200</mml:mn><mml:mspace width="0.16em"/><mml:mi>GeV</mml:mi></mml:mrow></mml:math> proton-nucleus collisions
  3. PHENIX results on jet modification with π0- and photon-triggered two particle correlations in p+p, p(d)+Au, and Au+Au collisions
  4. PHENIX results on jet modification with π0- and photon-triggered two particle correlations in p+p, p(d)+Au, and Au+Au collisions
  5. Speaker-Independent Acoustic-to-Articulatory Speech Inversion
  6. The buckeye corpus of speech: updates and enhancements
  7. Multi-Corpus Acoustic-to-Articulatory Speech Inversion
  8. Noise Robust Acoustic to Articulatory Speech Inversion
  9. Quantifying the Hyperparameter Sensitivity of Neural Networks for Character-level Sequence-to-Sequence Tasks
  10. MolRep: A Deep Representation Learning Library for Molecular Property Prediction
  11. Overtuning in Hyperparameter Optimization
  12. Toward Zero Oracle Word Error Rate on the Switchboard Benchmark