Dialogue Understandability: Why are we streaming movies with subtitles?

Helard Becerra Martinez, Alessandro Ragano, Diptasree Debnath, Asad Ullah, Crisron Rudolf Lucas, Martin Walsh, Andrew HinesarXiv 2024 · DOI · arXiv

Abstract

As streaming becomes the primary mode of film consumption, a growing number of viewers—including those without hearing impairments—increasingly rely on subtitles to follow dialogue. This paper, authored by researchers at University College Dublin and Xperi Inc., introduces the concept of Dialogue Understandability (DU). DU extends beyond traditional speech intelligibility to encompass the cognitive ease with which a viewer follows a narrative. By mapping DU factors across the media lifecycle and testing existing objective metrics, the study reveals a significant gap between current AI capabilities and the complexities of modern cinematic soundscapes, where artistic intent often conflicts with technical clarity.

Introduction: The Subtitle Phenomenon

Traditional audio engineering has long focused on two primary metrics: intelligibility (the ability to identify specific words) and quality (the technical clarity of the signal). However, these metrics fail to explain a burgeoning cultural trend: approximately 85% of Netflix users now utilize subtitles. This "subtitle crisis" suggests that while we may hear the words, we are struggling to understand the story without undue cognitive effort.

Timeline of audio format evolution, showing the increasing complexity from monaural to object-based spatial audio.

The researchers argue that this phenomenon is driven by a "perfect storm" of factors. Production trends like "mumble acting" prioritize realism over clarity, while modern sound mixing often results in a low Speech-to-Background Ratio (SBR), where music and sound effects overwhelm dialogue. Furthermore, consumption habits have shifted; audiences frequently watch complex content on mobile devices in noisy environments using subpar speakers, a far cry from the calibrated environments for which these soundtracks were originally designed.

Defining Dialogue Understandability (DU)

The paper seeks to formalize DU as a distinct field of study. Unlike intelligibility, DU accounts for the narrative context. For example, a viewer might achieve 100% intelligibility by hearing every word of a character with a thick accent, yet suffer from low understandability if the cognitive load required to process that accent distracts from the plot. DU is thus defined by the ease of following a story's plot without "undue cognitive effort."

The Conceptual Framework

To address this challenge, the authors developed a conceptual framework that maps DU factors across three Quality of Experience (QoE) domains: Human, System, and Context. This framework tracks these factors through the entire media lifecycle, from the initial creative decisions made by directors and actors to the final consumption by the end-user.

The DU framework mapping QoE factors across the media lifecycle, stakeholders, and measurement tools.

Methodology: Testing the "Messy" Reality

To evaluate whether DU can be measured automatically, the researchers conducted pilot experiments using several "off-the-shelf" objective tools on real-world, "messy" content rather than clean-speech datasets. Their test material included movie trailers (e.g., Casablanca, The Lord of the Rings) and multilingual/accented content (e.g., Squid Game, Narcos). The tools tested included:

  • SRMR: To measure non-intrusive intelligibility.
  • NISQA: A deep-learning model for predicting speech quality and noisiness.
  • ASR (Automatic Speech Recognition): Using Word Error Rate (WER) as a proxy for human cognitive load.
  • Language/Accent Classifiers: To test AI's ability to identify challenging dialects.

Key Insights: The Artistic Intent Conflict

A primary finding is the inherent conflict between technical metrics and artistic intent. Standard speech quality tools often flag intentional creative choices—such as Darth Vader’s processed voice or a whispered, high-tension scene—as "poor quality" or "noisy." Current AI lacks the context to distinguish between a technical failure and a deliberate stylistic decision. This highlights the SBR bottleneck: the massive dynamic range of modern cinema, where explosions are exponentially louder than whispers, is optimized for theaters but fails in home or mobile environments.

Results and Future Directions

The pilot studies demonstrated that objective metrics like SRMR and NISQA could not fully characterize perceived DU scores provided by human experts. Correlation was weak because these tools are not trained on cinematic audio. Additionally, ASR models failed significantly when background music was present, and language models frequently misidentified accents in entertainment contexts.

Future Work: The authors emphasize the need for "Movie-Aware" AI that can account for background noise and artistic distortion. Potential applications include "smart subtitling," where subtitles only appear during difficult-to-understand scenes, and "personalized dialogue boost," utilizing object-based audio to allow users to manually increase the dialogue track relative to music and effects.

Summary updated

Abstract

Watching movies and TV shows with subtitles enabled is not simply down to audibility or speech intelligibility. A variety of evolving factors related to technological advances, cinema production and social behaviour challenge our perception and understanding. This study seeks to formalise and give context to these influential factors under a wider and novel term referred to as Dialogue Understandability. We propose a working definition for Dialogue Understandability being a listener's capacity to follow the story without undue cognitive effort or concentration being required that impacts their Quality of Experience (QoE). The paper identifies, describes and categorises the factors that influence Dialogue Understandability mapping them over the QoE framework, a media streaming lifecycle, and the stakeholders involved. We then explore available measurement tools in the literature and link them to the factors they could potentially be used for. The maturity and suitability of these tools is evaluated over a set of pilot experiments. Finally, we reflect on the gaps that still need to be filled, what we can measure and what not, future subjective experiments, and new research trends that could help us to fully characterise Dialogue Understandability.
BibTeX
@misc{martinez2024dialogue,
  author = {Helard Becerra Martinez and Alessandro Ragano and Diptasree Debnath and Asad Ullah and Crisron Rudolf Lucas and Martin Walsh and Andrew Hines},
  title = {{Dialogue Understandability: Why are we streaming movies with subtitles?}},
  year = {2024},
  eprint = {2403.15336},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.15336},
}
Figure