Correlation and Interpolation Networks for Real-time Expression Analysis/Synthesis

Trevor Darrell, Irfan A. Essa, Alex PentlandNeurIPS 1994 · OpenReview

Introduction

This paper presents a pioneering framework for real-time facial expression tracking and synthesis. By mapping 2D image correlation scores to 3D muscle activations using Radial Basis Function (RBF) networks, the authors successfully combined fast, view-based perception with a physically constrained 3D model. The result was markerless facial animation achieved at interactive speeds—a significant milestone for its time.

The Challenge: 2D Speed vs. 3D Accuracy

In the early 1990s, analyzing facial expressions in real-time was a formidable challenge for computer vision. Traditional methods generally fell into two distinct camps, each with significant drawbacks:

  • Purely 2D methods: These approaches were fast but often relied on simple image warping or optical flow. Consequently, they could easily produce "impossible" facial configurations that looked unnatural or distorted.
  • Purely 3D methods: These utilized anatomically accurate models but were computationally expensive. Furthermore, they almost always required physical markers (dots or face paint) on the user's face to track movement reliably.

The authors sought a "middle way": a system fast enough for interactive applications like video conferencing or animation, completely markerless, yet still benefiting from the realistic constraints of a 3D anatomical model.

Methodology: A Hybrid Architecture

To achieve this balance, the authors proposed a hybrid architecture consisting of three main components: view-based perception, an RBF interpolation network, and a physically-based 3D model.

Face tracking and bounding box localization

1. View-Based Perception Instead of attempting to reconstruct a complex 3D face directly from a 2D image, the system uses a "distributed representation." During a training phase, it learns a set of iconic 2D templates (views) through a simple clustering algorithm. When a new video frame arrives, the system calculates a normalized correlation score between the input image and these stored views.

Examples of iconic 2D view templates

2. RBF Interpolation Network The "perceptual state"—defined as the vector of correlation scores from the view-based perception step—is then mapped to a "motor state" representing 3D muscle activations. This mapping is handled by a Radial Basis Function (RBF) network, which allows the system to interpolate smoothly and continuously between known expressions.

3. Physically-Based 3D Model Finally, the output of the RBF network drives an anatomically derived 3D mesh. This model simulates actual skin dynamics and muscle actuators. By routing the animation through this physical simulation, the system ensures that the resulting facial movements remain physically plausible, acting as a structural "sanity check" against noisy 2D tracking data.

To handle complex, independent facial movements—such as blinking one eye while smiling—the authors introduced decoupled templates. Instead of relying on one monolithic template for the entire face, they utilized separate correlation networks for specific regions like the eyes, eyebrows, and mouth.

Training and Data

Because standard public benchmarks for this specific task were rare at the time, the system relies on custom data gathering:

  • User-Specific Training Sequences: A short video of a user performing basic expressions (e.g., neutral, smile, surprise) is captured to automatically extract the view templates via clustering.
  • Paired Exemplars: To train the RBF network, the system needs pairs of perceptual scores and their corresponding muscle activations. These are gathered either via a "model-driven" approach (where the user mimics the 3D model) or a "user-driven" approach (using slower, offline optical flow methods to estimate muscle states from natural movements).

Key Insights

  • Dimensionality Reduction: By converting a high-dimensional image (thousands of pixels) into a low-dimensional vector of correlation scores (usually about a dozen dimensions), the complex interpolation problem becomes computationally trivial.
  • 2D for Speed, 3D for Constraints: The 3D model acts as a vital safeguard. Even if the 2D tracking is slightly noisy or ambiguous, the physical constraints of the 3D mesh prevent the synthesized face from breaking into unnatural poses.
  • Iconic Representation: Using literal image templates rather than abstract mathematical features allows the system to be robust across different users, provided a quick calibration step is performed.

Results and Performance

For interactive settings in 1994, the system's performance was highly impressive. It achieved a frame rate of over 5 Hz, which was considered real-time. This speed was reached using a dedicated single-board accelerator for the correlation math, paired with an SGI workstation for the 3D rendering. Crucially, the system successfully tracked and synthesized expressions like smiles and eye blinks robustly, without any special markers or makeup.

Limitations and Future Work

Despite its successes, the framework had several limitations:

  • User Calibration: The system is not immediately "plug-and-play" across different people; it requires a dedicated training sequence for each new user to build the view-based model.
  • Invariance Challenges: Handling translation and scale changes requires replicating units or exhaustive sampling, which is computationally expensive and necessitates hybrid search approximations.
  • Linear RBFs: The specific implementation used a simple linear ramp for the RBF. The authors noted that more complex basis functions might yield even smoother transitions.

Looking forward, the authors pointed toward promising applications in ultra-low bitrate video conferencing—where transmitting lightweight muscle activations instead of dense video frames could save massive amounts of bandwidth—and the development of highly personalized user interfaces.

Summary updated

Abstract

We describe a framework for real-time tracking of facial expressions that uses neurally-inspired correlation and interpolation methods. A distributed view-based representation is used to characterize facial state, and is computed using a replicated correlation network. The ensemble response of the set of view correlation scores is input to a network based interpolation method, which maps perceptual state to motor control states for a simulated 3-D face model. Activation levels of the motor state correspond to muscle activations in an anatomically derived model. By integrating fast and robust 2-D processing with 3-D models, we obtain a system that is able to quickly track and interpret complex facial motions in real-time.
BibTeX
@inproceedings{darrell1994correlation,
  author = {Trevor Darrell and Irfan A. Essa and Alex Pentland},
  title = {{Correlation and Interpolation Networks for Real-time Expression Analysis/Synthesis}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1994},
  url = {https://openreview.net/forum?id=Skbp2wWuZS},
}
Figure