Subjective Evaluation Methodologies for Speech and Audio Quality

95 concepts82 papers1 proposal

When evaluating speech and audio technologies—such as text-to-speech synthesis, voice clones, or audio codecs—traditional engineering metrics like root-mean-square error or signal-to-noise ratio fall short. Human hearing is highly non-linear, adaptive, and deeply tied to semantic comprehension. A packet loss artifact in a Voice over IP (VoIP) stream or a spectral distortion in a synthetic voice might register as a minor mathematical deviation, yet completely destroy the user experience.

To capture the true performance of these systems, researchers rely on subjective listening tests. However, human judgment is inherently variable, easily influenced by environmental context, listener background, and cognitive biases. The core challenge of subjective audio evaluation is transforming these fickle human impressions into reliable, reproducible scientific data.

Standardizing the Human Yardstick

The most common approach to quantifying audio quality is the Mean Opinion Score (MOS). In a standard Absolute Category Rating (ACR) setup, listeners hear an audio clip and rate its overall quality on a discrete 5-point scale. While straightforward, a 5-point scale can lack the resolution needed to differentiate high-performing systems. To capture finer perceptual nuances, such as the subtle degradations caused by network transmission errors, Anssi Rämö et al. utilized an extended 9-point scale (ACR9) in EVS Channel Aware Mode Robustness to Frame Erasures, providing a more granular view of voice quality under challenging network conditions.

However, a single global score often obscures why a system is failing. A synthetic voice might be highly intelligible but sound robotic and unpleasant, or it might sound natural but fail to articulate specific phonemes. To address this, frameworks like the ITU-T P.85 standard decompose quality into distinct perceptual dimensions. As demonstrated by Michael Pucher et al. in A user study on the influence of mobile device class, synthesis method, data rate and lexicon on speech synthesis quality and Dmitry Sityaev et al. in Comparison of the ITU-t p.85 standard to other methods for the evaluation of text-to-speech systems, listeners rate specific attributes independently:

  • Listening effort: How hard the listener must work to understand the speech.
  • Comprehension problems: Whether specific words or phrases are ambiguous.
  • Articulation and pronunciation: The mechanical clarity of the speech sounds.
  • Voice pleasantness: The aesthetic and natural quality of the speaker's tone.

This multi-dimensional profiling allows researchers to diagnose specific technical flaws. For instance, in Harnessing Text-to-Speech Voice Cloning Models for Improved Audiological Speech Assessment, Lidea Shahidi et al. adapted these dimensions to evaluate generative voice clones, isolating whether a model struggles with linguistic prosody or basic signal clarity.

Calibrating Listeners via Anchors and References

Human listeners suffer from range-equalizing bias. If an experiment only presents high-quality audio samples, listeners will stretch their ratings to use the full scale, artificially penalizing minor flaws. Conversely, if they only hear poor audio, mediocre samples look excellent. In Investigating Range-Equalizing Bias in Mean Opinion Score Ratings of Synthesized Speech, Erica Cooper et al. demonstrated that the specific context of audio samples significantly shifts absolute MOS ratings, making it difficult to compare different studies directly.

To anchor the listener's internal scale, researchers use referenced test paradigms. Instead of judging a clip in isolation, listeners compare it directly to a known baseline. The MUSHRA (MUltiple Stimuli with Hidden Reference and Anchor) methodology exemplifies this design. Originally designed for high-quality audio codecs, it has been widely adopted across spatial audio (Miroslaw Narbutt et al. in Streaming VR for immersion: Quality aspects of compressed spatial audio) and speech enhancement (Thomas Joubaud et al. in French Listening Tests for the Assessment of Intelligibility, Quality, and Identity of Body-Conducted Speech Enhancement).

MUSHRA Trial Structure:
[ Known Reference ] -> High-quality uncompressed audio
    ├── [ Stimulus A ] -> System under test
    ├── [ Stimulus B ] -> Hidden Reference (identical to known reference)
    └── [ Stimulus C ] -> Hidden Anchor (heavily degraded baseline)

In a MUSHRA trial, listeners are provided with an explicit, known high-quality reference. They then rate multiple anonymized stimuli on a continuous scale from 0 to 100. Crucially, the stimuli include a hidden reference (the pristine original) and at least one hidden anchor (a systematically degraded version, such as a low-pass filtered or robotized copy).

By forcing listeners to evaluate these control samples alongside the test systems, the methodology calibrates the scale across all participants. As Lionel Feugère et al. argue in Evaluation of Singing Synthesis: Methodology and Case Study with Concatenative and Performative Systems and Linda Bakkouche et al. show in Finding the Human Voice in AI: Insights on the Perception of AI-Voice Clones from Naturalness and Similarity Ratings, incorporating these explicit distortions provides a stable benchmark, ensuring that scores remain comparable across different research efforts and generative models.

Scaling Evaluations and Real-Time Feedback

Historically, listening tests required controlled laboratory environments and expert listeners. To scale evaluations, the field has shifted toward web-based crowdsourcing and real-time feedback mechanisms.

Crowdsourcing and Quality Control

Crowdsourcing platforms offer massive, diverse listener pools, but they introduce challenges like listener distraction, poor playback hardware, and bad-faith actors. In Crowdsourcing preference tests, and how to detect cheating, Sabine Buchholz et al. developed statistical metrics to automatically detect and exclude cheaters in speech synthesis preference tests.

Furthermore, porting complex frameworks like MUSHRA to the crowd requires strict procedural modifications. In Crowdsourcing MUSHRA Tests in the Age of Generative Speech Technologies: A Comparative Analysis of Subjective and Objective Testing Methods, Laura Lechler et al. demonstrated that while crowdsourced non-experts can reliably rank generative speech codecs, the platform design must include participant pre-screening, mandatory headphone validation tests, and strict limits on audio duration to mitigate listener fatigue.

Continuous, Real-Time Feedback

A major limitation of traditional MOS or MUSHRA tests is that they evaluate short, isolated sentences. When evaluating long, continuous speech—such as an entire audiobook or a lecture—a single post-hoc score fails to capture where specific errors occurred.

To solve this, Christina Tånnander et al. introduced a web-based Audience Response System (ARS) in Revisiting Three Text-to-Speech Synthesis Experiments with a Web-Based Audience Response System. Instead of rating the audio after it finishes, participants listen to continuous streams and click a designated interface area the exact moment they encounter an undesirable artifact or unnatural phrasing. This methodology provides a time-aligned, diagnostic map of speech quality, pinpointing transient synthesis errors that a global score would smooth over.

Algorithmic Approximations of Human Perception

While human listening tests are the gold standard, they are too slow and expensive to embed directly into a model's training loop or a live network router. This necessitates objective perceptual metrics—algorithms engineered to predict how a human would rate the audio.

An early but resilient approach is Mel-Cepstral Distortion (MCD), which calculates the mathematical distance between the spectral envelopes of synthesized and natural speech. In A Comparison of Speaker-based and Utterance-based Data Selection for Text-to-Speech Synthesis, Kai-Zhan Lee et al. showed that MCD maintains a strong correlation with human intelligibility evaluations, making it a valuable tool for rapid pre-selection when human testers are unavailable.

For complex transmission environments and modern audio codecs, researchers deploy sophisticated perceptual models like PESQ (Perceptual Evaluation of Speech Quality) and ViSQOL (Virtual Speech Quality Objective Listener). These algorithms align a degraded output signal with its pristine reference, passing both through a model of the human auditory system to calculate a predicted MOS.

For instance, Nate Kushman et al. utilized PESQ in Can you hear me now?!: it must be BGP to approximate the human perceptual impact of Border Gateway Protocol updates on cross-domain VoIP calls. Similarly, Kemal Altwlkany et al. leveraged ViSQOL in On the Language and Gender Biases in PSTN, VoIP and Neural Audio Codecs to systematically evaluate how different audio transcoding pipelines introduce demographic biases. By simulating human auditory judgment, these objective metrics allow researchers to optimize speech technologies at scale before final validation with human listener cohorts.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_subjective_audio_quality_assessment_a3c91d47,
  title = {subjective audio quality assessment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/subjective-audio-quality-assessment/rdr_a3c91d47702345858b6d9f883b81d6f0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_subjective_audio_quality_assessment_a3c91d47_v1,
  title = {subjective audio quality assessment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/subjective-audio-quality-assessment/rdr_a3c91d47702345858b6d9f883b81d6f0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake