A Data-Driven Investigation of Noise-Adaptive Utterance Generation with Linguistic Modification

Anupama Chingacham, Vera Demberg, Dietrich Klakow

Introduction

Spoken Dialog Systems (SDS), such as virtual assistants and in-car navigation systems, frequently encounter performance degradation in noisy environments. Traditionally, researchers have addressed this challenge through acoustic modifications—techniques like increasing volume, slowing speech rates, or synthesizing "Lombard speech" (the natural human tendency to increase vocal effort in noise). While effective, these methods often introduce signal processing artifacts or result in robotic, unnatural audio.

This paper explores a promising alternative: linguistic modification. By leveraging the fact that certain phonetic structures and word choices are inherently more robust to noise than others, the authors investigate whether paraphrasing a message can improve intelligibility without altering the acoustic signal's processing chain. Their findings suggest that intelligent paraphrase selection can improve human understanding by up to 33% in high-noise scenarios.

Methodology: The PiN Dataset and Human Perception

To systematically study the impact of linguistic choice, the researchers developed the Paraphrases in Noise (PiN) dataset. This benchmark was constructed through a multi-step process:

  1. Paraphrase Generation: Using a pre-trained PEGASUS model fine-tuned on paraphrase datasets, the authors generated multiple variations of short sentences sourced from the Switchboard corpus.
  2. Human Transcription Study: The researchers recruited 90 native English speakers to participate in a perception experiment. Participants listened to these paraphrases overlaid with "babble noise" (simulating a crowded room) at three distinct Signal-to-Noise Ratios (SNRs): 5 dB, 0 dB, and -5 dB.
  3. Quantifying Intelligibility: The study utilized a metric called Sentence-Level Intelligibility (Sent-Int), calculated based on the phoneme recognition rate. This measures how many individual sounds a listener correctly identifies, providing a more granular view of understanding than word-error rates alone.

PiN represents the largest available corpus of its kind, featuring 900 paraphrase pairs and comprehensive human transcription data across varying noise levels.

Modeling Noise-Robust Speech

The core of the research involves training a model to predict which paraphrase will be most intelligible in a given environment. The authors employed an SVMRank model to perform pairwise ranking of paraphrases. The model was trained using three primary feature sets:

  • Phoneme Length (phLen): The total number of phonemes in the utterance.
  • Perplexity (ppl): A measure of linguistic predictability derived from a language model.
  • STOI (Short-Time Objective Intelligibility): An acoustic metric that estimates how well the speech signal survives noise interference.

By combining these features, the model attempts to identify the "noise-robust acoustic cues" inherent in certain linguistic structures.

Key Results and Insights

The experimental results highlight a significant untapped potential for improving SDS robustness through language:

  • The Oracle Potential: If a system could perfectly select the most intelligible paraphrase every time (the "oracle" scenario), intelligibility would increase by 33% relative at the harshest noise level (-5 dB).
  • Predictive Performance: The proposed ranking model achieved a 31.37% relative improvement over a uniform baseline at -5 dB, demonstrating that linguistic features like length and predictability are strong indicators of noise robustness.
  • Brevity and Predictability: The data suggests that shorter utterances (fewer phonemes) and more predictable phrasing (lower perplexity) generally perform better. Shorter sentences offer less "surface area" for noise to corrupt, while predictable language allows listeners to mentally "fill in the blanks" when the audio is partially obscured.
  • Noise Sensitivity: The benefits of paraphrasing are most pronounced in high-noise environments. At 5 dB SNR, a "ceiling effect" occurs where most speech is understood regardless of phrasing; however, as the SNR drops, the choice of words becomes critical for communication success.

Limitations and Future Directions

While the study proves the efficacy of linguistic modification, several areas remain for future exploration. The current research focused exclusively on babble noise; different noise types (e.g., white noise or engine hums) may interact differently with phonetic structures. Additionally, the study used "quasi-paraphrases" to ensure a wide variety of candidates, which can sometimes lead to subtle shifts in semantic nuance.

Future work will likely focus on maintaining strict semantic equivalence while maximizing clarity, as well as integrating real-time noise estimation to allow SDS to decide dynamically when a paraphrase is necessary. By moving beyond simple volume adjustments and toward "saying it differently," we can create voice interfaces that remain clear and reliable in the most challenging acoustic environments.

Summary updated

BibTeX
@inproceedings{chingacham2022data,
  author = {Anupama Chingacham and Vera Demberg and Dietrich Klakow},
  title = {{A Data-Driven Investigation of Noise-Adaptive Utterance Generation with Linguistic Modification}},
  booktitle = {SLT},
  year = {2022},
  doi = {10.1109/slt54892.2023.10022437},
}
Figure