Relating Objective and Subjective Performance Measures for AAM-Based Visual Speech Synthesis

Barry-John Theobald, Iain A. MatthewsIEEE Trans. Speech Audio Process 2012 · OpenReview

Abstract

We compare two approaches for synthesizing visual speech using active appearance models (AAMs): one that utilizes acoustic features as input, and one that utilizes a phonetic transcription as input. Both synthesizers are trained using the same data and the performance is measured using both objective and subjective testing. We investigate the impact of likely sources of error in the synthesized visual speech by introducing typical errors into real visual speech sequences and subjectively measuring the perceived degradation. When only a small region (e.g., a single syllable) of ground-truth visual speech is incorrect we find that the subjective score for the entire sequence is subjectively lower than sequences generated by our synthesizers. This observation motivates further consideration of an often ignored issue, which is to what extent are subjective measures correlated with objective measures of performance? Significantly, we find that the most commonly used objective measures of performance are not necessarily the best indicator of viewer perception of quality. We empirically evaluate alternatives and show that the cost of a dynamic time warp of synthesized visual speech parameters to the respective ground-truth parameters is a better indicator of subjective quality.

Introduction

In the field of visual speech synthesis—the creation of digital faces that "speak" in synchronization with audio—evaluation typically relies on two pillars: objective mathematical metrics and subjective human judgment. Objective measures are favored for being inexpensive and repeatable, often serving as a proxy for human opinion. However, there has been limited validation regarding whether a "good" score on paper translates to a natural experience for a human viewer. This research addresses this gap by comparing two distinct synthesis strategies and analyzing how various mathematical metrics correlate with human perception.

Methodology

The study utilizes Active Appearance Models (AAMs), which represent a face through a combination of shape (geometry) and appearance (pixel texture). Two primary synthesis architectures were evaluated:

  1. Feature-driven Synthesis: A neural network maps acoustic features (MFCCs) directly to visual parameters on a frame-by-frame basis.
  2. Unit-driven Synthesis: A concatenative approach that selects and stitches together segments of real recorded visual speech based on phonetic context.
Comparison of Feature-driven and Unit-driven synthesis architectures.

The researchers used the Messiah corpus, containing 279 sentences from a single speaker captured at 25 frames per second. A leave-one-out cross-validation method ensured the synthesizers were tested on novel sentences.

Evaluation Metrics

To bridge the gap between math and perception, five objective metrics were tested against human scores:

  • Correlation (ρ): Measures how well predicted and real trajectories move together.
  • Normalized RMS Error (ε): The average distance between predicted and real parameters.
  • Normalized Peak RMS Error (εp): The single largest error found in a sequence.
  • Dynamic Time Warping (DTW) Cost (D): A measure accounting for both distance and temporal alignment.
  • Mahalanobis Distance (d): A phone-based distance measure weighting errors by the variance of specific sounds.

Results and Discussion

The "One Bad Apple" Effect

A critical finding was that humans are exceptionally sensitive to localized errors. By introducing an error into just a single syllable of ground-truth speech, the researchers observed that the naturalness score for the entire sentence plummeted. This explains why standard metrics like RMS error fail; they average out these critical, short-lived failures across the entire sequence.

Example of a synthesized face using an Active Appearance Model (AAM).

Metric Performance

The study revealed that DTW cost (D) was the most reliable objective indicator of subjective quality, with a correlation of 0.76 for shape parameters. Because DTW considers the temporal relationship between signals rather than just frame-by-frame distance, it better captures the "flow" of speech. Conversely, the widely used trajectory correlation (ρ) showed a shockingly low relationship with human perception for appearance parameters (0.20).

Synthesis Preference

Subjective testing showed a clear preference for the unit-driven approach. On a 0–50 scale, re-rendered ground truth scored 38.71, unit-driven synthesis scored 27.47, and feature-driven synthesis scored 22.35. The unit-driven approach likely succeeds because it preserves natural co-articulation—the subtle way one sound influences the next—found in original recordings.

Conclusion

This research highlights a significant disconnect between standard objective metrics and human perception in visual speech synthesis. While RMS error and correlation are common, they are poor predictors of quality. Instead, metrics that account for temporal alignment (DTW) and peak errors are far more effective. Future work should focus on developing "super-metrics" that combine these measures to better simulate human judgment and improving unit selection techniques to minimize the need for heavy post-processing smoothing, which often leads to mumbly, under-articulated animations.

Summary updated

Abstract

We compare two approaches for synthesizing visual speech using active appearance models (AAMs): one that utilizes acoustic features as input, and one that utilizes a phonetic transcription as input. Both synthesizers are trained using the same data and the performance is measured using both objective and subjective testing. We investigate the impact of likely sources of error in the synthesized visual speech by introducing typical errors into real visual speech sequences and subjectively measuring the perceived degradation. When only a small region (e.g., a single syllable) of ground-truth visual speech is incorrect we find that the subjective score for the entire sequence is subjectively lower than sequences generated by our synthesizers. This observation motivates further consideration of an often ignored issue, which is to what extent are subjective measures correlated with objective measures of performance? Significantly, we find that the most commonly used objective measures of performance are not necessarily the best indicator of viewer perception of quality. We empirically evaluate alternatives and show that the cost of a dynamic time warp of synthesized visual speech parameters to the respective ground-truth parameters is a better indicator of subjective quality.
BibTeX
@article{theobald2012relating,
  author = {Barry-John Theobald and Iain A. Matthews},
  title = {{Relating Objective and Subjective Performance Measures for AAM-Based Visual Speech Synthesis}},
  journal = {IEEE Trans. Speech Audio Process.},
  year = {2012},
  url = {https://openreview.net/forum?id=rBBWJdnXfm5},
}
Figure