Synthetic Audio Helps for Cognitive State Tasks

Adil Soubki, John Murzaku, Peter Zeng, Owen RambowNAACL 2025 · DOI

Introduction

In human communication, the meaning of an utterance is often shaped as much by how it is said as by the words themselves. Paralinguistic cues—such as prosody, rhythm, and intonation—convey critical information about a speaker's emotional state, certainty, and intent. While modern Natural Language Processing (NLP) has achieved remarkable success in text analysis, it frequently overlooks these rich acoustic signals. This is largely due to the scarcity of high-quality multimodal datasets; most large-scale datasets for "cognitive state" tasks (e.g., predicting sentiment or belief) are text-only.

Researchers from Stony Brook University address this gap by proposing that models can "hallucinate" missing audio information. Their framework, Synthetic Audio Data (SAD) fine-tuning, demonstrates that generating synthetic audio from text using Text-to-Speech (TTS) engines can significantly enhance a model's performance on cognitive state tasks.

The SAD Framework

The SAD framework leverages the "priors" embedded in modern TTS systems. When a high-quality TTS model generates speech, it must make decisions about prosody and emphasis to sound natural. These decisions provide an "orthogonal signal"—a set of features mathematically independent of text patterns—that can aid a classifier in understanding cognitive context.

The Synthetic Audio Data (SAD) fine-tuning pipeline.

The pipeline consists of four main stages:

  1. Synthesis: Text snippets are converted to audio using zero-shot TTS models, such as OpenAI’s TTS API (using the 'Alloy' voice) or the open-source Matcha-TTS.
  2. Encoding: The text is processed by a text encoder (BERT-base-uncased), while the synthetic audio is processed by an audio encoder (OpenAI’s Whisper-base).
  3. Fusion: The resulting representations are combined using early fusion (concatenation) or late fusion (combining separate head outputs).
  4. Fine-tuning: The entire multimodal stack is fine-tuned on the target cognitive task.

Experimental Evaluation

The authors evaluated SAD across seven cognitive state tasks and three control tasks from the SuperGLUE benchmark. The tasks were categorized into:

  • Sentiment: SWBD-S (with gold audio) and IMDB (text-only).
  • Belief/Factuality: CB-Prosody (with gold audio), CommitmentBank (CB, text-only), and FactBank (text-only).
  • Emotion: IEMOCAP (with gold audio) and GoEmotions (text-only).
  • Control: BoolQ, WiC, and WSC (logic-based tasks not expected to benefit from prosody).

Key Results

The experimental results, summarized in the table below, confirm that synthetic audio provides a meaningful performance boost for cognitive tasks.

Performance comparison across various tasks and modalities.

Synthetic vs. Gold Audio

For tasks where human "gold" audio was available, synthetic audio proved to be a surprisingly effective proxy. On the CB-Prosody belief dataset, using OpenAI-generated synthetic audio led to a 3.6% reduction in Mean Absolute Error (MAE), nearly matching the 4% reduction achieved with real human speech. However, for complex emotion recognition (IEMOCAP), gold audio still maintained a significant lead (63.4 F1 vs. 59.3 F1 for synthetic), suggesting that current TTS models struggle to capture the full spectrum of authentic human emotion.

Enhancing Text-Only Tasks

SAD consistently improved performance on datasets that lacked any original audio. For instance, on the FactBank belief dataset, the F1 score rose from 74.9 (text-only) to 76.0 (multimodal). Similarly, on the GoEmotions dataset, the F1 score increased from 51.4 to 53.1. These improvements were not observed in the control tasks (e.g., WSC), confirming that the framework specifically captures expressive paralinguistic signals rather than acting as a generic regularizer.

Discussion and Future Work

The success of SAD highlights that TTS models implicitly encode cognitive information during their training for naturalness. While higher-quality closed models like OpenAI's generally outperformed open-source alternatives like Matcha-TTS, both provided gains over text-only baselines.

Future research will need to address the computational costs of large-scale audio generation and explore whether these benefits persist when using much larger language models. Additionally, investigating the impact of voice diversity—varying age, gender, and accent—could further enhance the robustness of multimodal cognitive state recognition.

Summary updated

BibTeX
@inproceedings{soubki2025synthetic,
  author = {Adil Soubki and John Murzaku and Peter Zeng and Owen Rambow},
  title = {{Synthetic Audio Helps for Cognitive State Tasks}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.92},
}
Figure