Speaker-following Video Subtitles

Yongtao Hu, Jan Kautz, Yizhou Yu, Wenping WangACM TOMM 2014 · DOI · OpenReview

Abstract: We propose a new method for improving the presentation of subtitles in video (e.g. TV and movies). With conventional subtitles, the viewer has to constantly look away from the main viewing area to read the subtitles at the bottom of the screen, which disrupts the viewing experience and causes unnecessary eyestrain. Our method places on-screen subtitles next to the respective speakers to allow the viewer to follow the visual content while simultaneously reading the subtitles. We use novel identification algorithms to detect the speakers based on audio and visual information. Then the placement of the subtitles is determined using global optimization. A comprehensive usability study indicated that our subtitle placement method outperformed both conventional fixed-position subtitling and another previous dynamic subtitling method in terms of enhancing the overall viewing experience and reducing eyestrain.

The Physiological Challenge of Subtitles

Conventional video subtitles are almost universally fixed at the bottom of the screen. While this is a long-standing standard, it creates a significant physiological challenge for the viewer. The human "vision span"—the area of the visual field where text can be read clearly—is limited to approximately 6 degrees of arc.

The geometry of the human vision span relative to a screen, illustrating why bottom-placed subtitles cause eyestrain.

When subtitles are placed at the bottom edge, viewers must constantly shift their gaze between the actors' faces in the center of the frame and the text at the periphery. This repetitive "ping-pong" eye movement leads to significant eyestrain and prevents viewers from fully absorbing the visual performance and facial expressions of the actors. To address this, we propose a system that automatically places subtitles directly next to the active speaker.

System Architecture

Implementing speaker-following subtitles automatically requires solving two primary technical challenges: accurately identifying the active speaker in potentially crowded or complex scenes and placing the text so it is readable without obscuring critical visual information. Our framework addresses these through a two-stage process: Speaker Detection and Smart Subtitle Placement.

The system pipeline, showing the flow from input video and subtitle files through speaker detection and placement optimization.

1. Multi-Modal Cascade Detection

Rather than relying on a single feature, our system utilizes a "degenerate decision tree" or cascade to identify the speaker. This approach is computationally efficient, as it rejects non-speakers using simpler checks before proceeding to more intensive analysis:

  • Face & Clothing Tracking: We use Viola-Jones for face detection but supplement it with clothing appearance tracking. This allows the system to maintain a speaker's identity even when their face is blurred, turned away, or temporarily occluded.
  • Lip Motion Analysis: We measure the Mean Squared Difference (MSD) of pixels in the mouth region to detect active speech.
  • Cinematographic Cues: The system leverages the "center contribution" rule, recognizing that speakers are typically positioned toward the center of the frame.
  • Audio-Visual (AV) Synchrony: The final stage correlates audio signals with refined motion regions based on facial landmarks to confirm the speaker's identity.

2. Subtitle Placement Optimization

To determine the optimal position for the text, the system minimizes a total energy function composed of three distinct terms:

  • Local Energy: Ensures the subtitle is close to the speaker but does not overlap with other faces.
  • Global Energy: Maintains positional consistency across consecutive frames to prevent distracting "jitter."
  • Layout Energy: Penalizes placement near screen boundaries to keep the viewer’s focus centralized.

Evaluation and Results

We evaluated the system using clips from popular media, including Friends, The Big Bang Theory, and films like Erin Brockovich and Up in the Air. Our multi-modal detection method improved speaker identification accuracy by 7.5% to 20.5% over previous state-of-the-art techniques.

In a usability study involving 219 participants, our "Smart Subtitle" system consistently scored higher for overall viewing experience and reduced eyestrain compared to both traditional static subtitles and previous dynamic placement methods. Participants noted that the system allowed them to stay more engaged with the actors' performances. While the system currently faces challenges with off-screen voices or extreme occlusions (such as masks), it represents a significant step toward a more natural and comfortable viewing experience for subtitled content.

Summary updated

Abstract

We propose a new method for improving the presentation of subtitles in video (e.g. TV and movies). With conventional subtitles, the viewer has to constantly look away from the main viewing area to read the subtitles at the bottom of the screen, which disrupts the viewing experience and causes unnecessary eyestrain. Our method places on-screen subtitles next to the respective speakers to allow the viewer to follow the visual content while simultaneously reading the subtitles. We use novel identification algorithms to detect the speakers based on audio and visual information. Then the placement of the subtitles is determined using global optimization. A comprehensive usability study indicated that our subtitle placement method outperformed both conventional fixed-position subtitling and another previous dynamic subtitling method in terms of enhancing the overall viewing experience and reducing eyestrain.
BibTeX
@article{hu2014speaker,
  author = {Yongtao Hu and Jan Kautz and Yizhou Yu and Wenping Wang},
  title = {{Speaker-following Video Subtitles}},
  journal = {ACM Trans. Multim. Comput. Commun. Appl.},
  year = {2014},
  doi = {10.1145/2632111},
}
Figure