Auditory-Feedback-Conditioned Flow TTS for Implicit Lombard Adaptation

Yunya Wang, Satoshi NakamuraarXiv 2026 · arXiv

TL;DR

Researchers have developed a way for Text-to-Speech (TTS) systems to "hear" environmental noise and automatically adjust their speaking style—much like humans do via the Lombard effect. By injecting real-time SNR and ASR feedback into a Flow-based TTS model, the system learns to improve its own intelligibility in noisy environments without needing explicit instructions to "speak louder" or "raise pitch."

What Problem Does This Paper Address?

Most conventional TTS systems are "deaf" to their surroundings. They generate high-quality speech in a vacuum, but if you play that speech in a noisy room, it quickly becomes unintelligible. Humans solve this naturally through the Lombard effect: when we hear noise, we instinctively increase our vocal effort, pitch, and intensity to be heard.

While previous research has tried to mimic this in AI, it usually relied on "explicit control"—manually telling the model to increase volume or pitch by a certain amount. This paper asks a more ambitious question: Can a modern, Flow-based TTS model learn to adapt implicitly? By creating a "perception-generation" loop where the model receives feedback on how noisy the environment is and how hard its own speech is to recognize, can it figure out the best acoustic response on its own?

Methods

The authors build their system on CosyVoice2, a state-of-the-art architecture consisting of a Large Language Model (LLM), a Flow-matching acoustic model, and a HiFT vocoder.

The core innovation is the injection of two types of Auditory Feedback into the Flow-matching condition:

  1. SNR Feedback (via SNRNet): A pretrained network estimates the noise level (Clean, 0 dB, or -10 dB) from the audio. This gives the model a sense of "how loud" the background noise is.
  2. ASR Feedback (via Whisper): The system passes its own noise-corrupted output through a frozen Whisper ASR model. By calculating the "recognition difficulty" (cross-entropy loss) of the tokens, the model gets a signal of "how confusing" its speech sounds to a listener.

These feedback signals are projected into the Flow conditioning space. Crucially, the model isn't given explicit targets for pitch or power; it must learn to map these feedback signals to the appropriate acoustic adjustments during training. The authors tested both static inference (refining a whole utterance) and dynamic inference (updating the voice block-by-block in a streaming fashion).

Datasets

  • Training: The Audio-Visual Lombard Grid corpus, which contains paired "plain" and "Lombard" (speech in noise) recordings from 54 speakers.
  • Evaluation: To test how well the model generalizes, the researchers used the Wall Street Journal (WSJ) corpus and the Hurricane dataset.
  • Noise Types: They used Gaussian white noise for training and the WHAM! dataset (real-world ambient noise) to test generalization to unseen noise types.

Key Insights and Ideas

  • The Power of Implicit Learning: The model successfully learned to adapt its voice without explicit prosodic labels. This suggests that modern Flow-based models are flexible enough to infer complex acoustic transformations directly from high-level feedback.
  • Feedback vs. Fine-tuning: While simply fine-tuning a model on Lombard speech makes it robust to noise it has seen before, auditory feedback provides a significant edge when encountering unseen noise levels or types. It acts as a real-time "sanity check" for the generator.
  • The "Tracking" Paradox: One of the most surprising findings was that "stronger tracking does not equal better quality." A model might perfectly track noise levels by increasing its volume (high correlation), but that doesn't mean the resulting speech sounds natural or is easier to understand. The joint SNR+ASR feedback provided the best balance, even if its raw "tracking" numbers weren't the highest.
  • Cue-Specific Benefits: SNR feedback was best at handling volume/environment following, while ASR feedback was more effective at improving intelligibility (STOI) across different noise levels.

Key Results

  • Intelligibility (STOI): Under static noise, ASR feedback consistently improved Short-Time Objective Intelligibility (STOI) across all tested Signal-to-Noise Ratios compared to a standard fine-tuned baseline.
  • Word Error Rate (WER): In unseen noise conditions (e.g., -5 dB), the joint SNR+ASR model significantly reduced the WER, showing better generalization than the baseline.
  • Subjective Quality (MOS): In listening tests, the joint SNR+ASR feedback model received the highest Mean Opinion Scores for both naturalness and how appropriately the speech adapted to dynamic noise.
  • Latency: The ASR feedback update (using Whisper) takes roughly 214ms to 298ms, making it potentially viable for near-real-time applications, though it adds computational overhead.

Limitations and Future Work

  • Computational Cost: Running a full ASR model (Whisper) in the feedback loop is resource-intensive. Future work could look into lighter-weight "proxy" models to provide recognition feedback.
  • The "Deaf" Gap: While the model adapts to noise, it still relies on a first-pass generation that might be "clean." The authors suggest that tighter integration of the perception loop during the very first stages of synthesis could yield even better results.
  • In-Domain Dominance: Under very specific, known noise conditions, simple fine-tuning is still incredibly hard to beat. The real value of this feedback approach lies in its ability to handle the "wild"—unpredictable, dynamic, and unseen acoustic environments.

Summary updated

Abstract

Conventional text-to-speech (TTS) systems can lose intelligibility in noise environments because they cannot adapt to environmental changes. However, humans tend to increase their vocal efforts in noisy environments, a phenomenon known as the Lombard effect. Building on an auditory-feedback Lombard TTS, we investigate whether a pretrained Flow-based TTS can learn acoustic adaptation directly from auditory feedback without explicit prosodic control. Signal-to-noise ratio (SNR) feedback and automatic speech recognition (ASR) feedback are extracted from the noise-corrupted waveforms and injected into the Flow-matching condition for utterance refinement and causal block-level updates. We assess feedback benefits across prompting and noise conditions using plain/Lombard prompting, prompt-switching controls, and paired-finetuning. Experiments show that while finetuning provides strong robustness under in-domain conditions, auditory feedback yields additional cue-dependent benefits that generalize to unseen noise. Moreover, stronger power/F0 tracking alone does not guarantee better speech intelligibility or quality.
BibTeX
@misc{wang2026auditory,
  author = {Yunya Wang and Satoshi Nakamura},
  title = {{Auditory-Feedback-Conditioned Flow TTS for Implicit Lombard Adaptation}},
  year = {2026},
  eprint = {2611.19260},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19260},
}
Figure