Scaling Neural Face Synthesis to High FPS and Low Latency by Neural Caching

Frank Yu, Sid Fels, Helge RhodinWACV 2023

TL;DR

This paper tackles the challenge of achieving real-time, low-latency neural rendering for 3D face synthesis, crucial for immersive applications like VR telepresence. They introduce a novel "neural caching" approach with an implicit warp layer that significantly reduces latency and increases frame rates by reusing information from previous frames, achieving near real-time performance with minimal quality degradation.

What Problem Does This Paper Address?

Modern neural rendering techniques can produce photorealistic images of faces, but their inherent sequential, layer-wise processing in deep neural networks leads to high runtime and latency. This makes them unsuitable for applications like virtual reality (VR) and telepresence, which demand extremely low latency (motion-to-photon latency) to provide a smooth and immersive user experience. Existing methods struggle to balance high image quality with the speed required for these interactive applications. The core problem is breaking the sequential dependency in deep neural networks to achieve both high frame rates and low latency without sacrificing visual fidelity.

Methods

The paper introduces a neural caching approach combined with an implicit warp layer.

  • Neural Caching: Instead of recomputing everything from scratch for each new frame, the system caches intermediate features (specifically, the last three layers of the generator network) from the previous frame. This cached information, along with relevant parameters like UV maps, camera pose, and expression coefficients, forms the "neural cache."
  • Implicit Warping: A shallow, two-layer neural network (W) is used to "warp" the cached information to generate the new frame. This warp is "implicit" because it doesn't rely on explicit geometric transformations but rather learns to approximate how the image changes with respect to pose and viewpoint. This shallow network is significantly faster than the full generator network.
  • Parallel Execution: The approach is designed to scale across multiple GPUs. For instance, one GPU can handle the generator and caching, while another handles the warp, or frames can be processed in a ping-pong fashion across GPUs. This parallelization primarily boosts frame rate, while the caching and implicit warp are key to reducing latency.
  • Head Stabilization: To combat jitter and flicker in real-world scenarios, they employ techniques like centering the virtual camera on the subject's head and using a delayed Gaussian filter for head position smoothing.

Datasets

The paper evaluates their method on several datasets:

  • Beard dataset: Used for offline evaluation, likely a custom dataset for testing facial hair rendering.
  • High-fps sequence: A dataset recorded at 60 fps, specifically for evaluating online performance.
  • Trump, Obama, male, and female datasets: These appear to be existing talking head datasets from previous work ([24]), used for comparison and general evaluation.

The paper does not introduce new benchmarks but rather uses existing datasets to demonstrate the effectiveness of their approach against prior work.

Key Insights and Ideas

  • Breaking Sequential Dependency is Key: The core insight is that the sequential nature of deep neural networks is the primary bottleneck for low latency. By caching and warping, they effectively create a "skip connection" that bypasses much of the sequential computation.
  • Shallow Warp Network for Latency: A surprisingly shallow network can effectively "warp" previous frame information for new viewpoints, demonstrating that complex geometric transformations aren't always necessary for good results in this specific domain.
  • Implicit Warping Outperforms Explicit: The paper found that their implicit warp, learned end-to-end, is more efficient and effective than explicit warping methods, which can be computationally expensive and ill-defined for occluded regions.
  • Parallelism for Frame Rate, Caching for Latency: They clearly distinguish how their approach addresses both frame rate (via multi-GPU parallelism) and latency (via caching and implicit warping), showing that these are complementary but distinct goals.
  • Minimal Quality Trade-off: The most impressive finding is achieving significant speedups with only a minor degradation in image quality, making it practical for real-world applications.

Key Results

  • Latency Reduction: Achieved a latency reduction of up to 70% (from 49.4ms to 14.9ms on commodity GPUs).
  • Frame Rate Improvement: Scaled frame rates accordingly, with improvements of up to 300% when using multiple GPUs.
  • Image Quality: Minimal degradation in image quality, reported as only 1% PSNR loss compared to baselines.
  • Runtime: The warp network has a runtime of 14.62ms, significantly faster than the generator backbone's 47.02ms.
  • Multi-GPU Synchronization Overhead: Negligible overhead of only 0.25ms/frame for synchronization between GPUs.

Limitations and Future Work

  • Training Data Dependency: The method requires a diverse set of training views to perform novel view synthesis and accurate warping. Extreme angles or occluded parts of the face (like ears) can lead to artifacts, especially if not well-represented in training.
  • Face Reconstruction Limitations: The performance is also limited by the underlying face reconstruction algorithm, which can struggle with occluded regions, eyes, and wide-open mouths.
  • Glasses and Expressions: The FLAME model used for facial representation has limitations in modeling glasses or complex expressions that are not explicitly represented.
  • Background Approximation: The current approach only approximates the background, leading to the use of static backgrounds in predictions.

Future work could focus on improving the robustness to extreme viewpoints, integrating more sophisticated face models, and extending the approach to handle dynamic backgrounds. The authors also suggest that their caching approach is compatible with deeper neural networks, paving the way for higher output resolutions.

Summary updated

BibTeX
@inproceedings{yu2023scaling,
  author = {Frank Yu and Sid Fels and Helge Rhodin},
  title = {{Scaling Neural Face Synthesis to High FPS and Low Latency by Neural Caching}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure