SurFhead: Affine Rig Blending for Geometrically Accurate 2D Gaussian Surfel Head Avatars

Jaeseong Lee, Taewoong Kang, Marcel C. Bühler, Min-Jung Kim, Sungwon Hwang, Junha Hyung, Hyojin Jang, Jaegul Choo

TL;DR

SurFhead is a new framework for creating high-fidelity, riggable head avatars using 2D Gaussian surfels and affine rigging. By replacing simple similarity transforms with Jacobian-based deformations and a novel "Jacobian Blend Skinning" technique, it achieves state-of-the-art geometric accuracy and handles extreme facial expressions without the "blobby" artifacts common in previous methods.

What Problem Does This Paper Address?

While recent head avatar models based on 3D Gaussian Splatting (3DGS) produce photorealistic images, their underlying geometry is often inaccurate. Most existing methods rely on similarity transformations—which only allow for rotation, translation, and uniform scaling—to move Gaussians along with a face mesh. However, human skin stretches and shears significantly during speech and emotion. Forcing these complex deformations into a rigid similarity framework leads to "over-smoothed" geometry and artifacts in extreme poses.

Furthermore, many models suffer from a "hollow illusion" in the eyes, where the cornea appears concave rather than convex. This happens because the optimization process prioritizes matching the image pixels (photometric loss) by making the eye surface transparent to simulate specularity, rather than reconstructing the actual curved geometry of the eyeball.

Methods

SurFhead introduces three key technical innovations to bridge the gap between classical graphics and modern neural rendering:

  1. 2D Gaussian Surfels with Affine Rigging: Instead of 3D "blobs," the authors use 2D disks (surfels). These have well-defined normals and depths. They move these surfels using a Jacobian deformation gradient ($J$), which allows for full affine transformations, including the stretch and shear essential for realistic skin movement. Crucially, they use the inverse-transpose of the Jacobian ($J^{-T}$) to transform surface normals, ensuring they remain mathematically consistent with the deformed surface.
  2. Jacobian Blend Skinning (JBS): To ensure smooth transitions between different parts of the face mesh, the authors propose JBS. Standard linear blending of transformation matrices can cause geometry to "collapse" or distort (see the "Direct Interpolation" in Figure 4). JBS uses Polar Decomposition to split the Jacobian into a rotation component ($U$) and a stretch/shear component ($P$). Rotations are blended in a specialized matrix-logarithm space, while stretch/shear is blended linearly, preventing geometric distortions.
Figure 2: Toy examples on Jacobian deformation and Polar Decomposition.
  1. Eyeball Regularization & ASGs: To fix the "hollow eye" problem, the authors constrain the eyeball region to be convex and opaque. They replace standard Spherical Harmonics (which are too low-frequency to capture sharp glints) with Anisotropic Spherical Gaussians (ASGs). This allows the model to render sharp, realistic specular reflections on the cornea without sacrificing geometric accuracy.
Figure 3: Overall pipeline of SurFhead. Only from RGB videos, SurFhead constructs geometri- cally accurate head avatars, equipped with our intricate deformations. The Jacobian J covers stretch and shear deformations avoiding surface distortion. Moreover, the blended Jacobian Jb alleviates inherent local deformations’ discontinuity. Finally, elaborated modeling of eyeballs such as preser- vation of specularity and convexity achieves more realistic appearance and geometry.

Datasets

The authors evaluated SurFhead on two primary datasets:

  • FaceTalk: A monocular synthetic dataset based on the FLAME model. It includes ground-truth surface normals, making it ideal for measuring geometric fidelity.
  • NeRSemble: A multi-view real-world dataset featuring 16 cameras and diverse facial expressions. The authors created a custom "held-out" set of extreme expressions to test how well the models generalize to difficult poses.

Key Insights and Ideas

  • Normals are the key to realism: By focusing on "geometrically accurate" rigging, the authors show that better normals lead to better relighting and more robust performance in extreme poses.
  • Classical Graphics meets 3DGS: The paper demonstrates that "old-school" techniques like Polar Decomposition and Jacobian gradients are still highly relevant for solving the stability issues of modern neural point-based representations.
  • The "Hollow Eye" is an optimization trap: The discovery that models "cheat" by using transparency to fake specularity highlights the need for explicit geometric constraints (like convexity) when dealing with highly reflective surfaces like eyes.

Key Results

SurFhead outperforms existing baselines like GaussianAvatars and SplattingAvatar across several metrics:

  • Geometric Accuracy: On the FaceTalk dataset, SurFhead achieved a Normal Cosine Similarity (NCS) of 0.983, significantly higher than baselines (e.g., PointAvatar at 0.756).
  • Rendering Quality: It maintains high PSNR and SSIM even in extreme poses where other methods produce semi-transparent or "blob-like" artifacts.
  • Efficiency: Despite the complex math, the model renders at roughly 90 FPS on a modern GPU, making it suitable for real-time applications.
Ablation study showing how Jacobian deformations and JBS reduce artifacts in extreme poses compared to the "Vanilla" approach.

Limitations and Future Work

The authors acknowledge a few remaining hurdles:

  • Expression Bounds: Because the model is tied to the FLAME mesh, it cannot represent expressions that fall far outside the FLAME expression space (e.g., very exaggerated faces that the tracker can't follow).
  • Missing Details: The current model lacks explicit representations for hair strands and the tongue, which are not part of the underlying 3D Morphable Face Model (3DMFM).
  • Computational Bottleneck: The Polar Decomposition step currently relies on SVD (Singular Value Decomposition), which can be slow for large batches. The authors suggest that specialized 3x3 matrix routines could further boost speed in the future.

Summary updated

Abstract

Recent advancements in head avatar rendering using Gaussian primitives have achieved significantly high-fidelity results. Although precise head geometry is crucial for applications like mesh reconstruction and relighting, current methods struggle to capture intricate geometric details and render unseen poses due to their reliance on similarity transformations, which cannot handle stretch and shear transforms essential for detailed deformations of geometry. To address this, we propose SurFhead, a novel method that reconstructs riggable head geometry from RGB videos using 2D Gaussian surfels, which offer well-defined geometric properties, such as precise depth from fixed ray intersections and normals derived from their surface orientation, making them advantageous over 3D counterparts. SurFhead ensures high-fidelity rendering of both normals and images, even in extreme poses, by leveraging classical mesh-based deformation transfer and affine transformation interpolation. SurFhead introduces precise geometric deformation and blends surfels through polar decomposition of transformations, including those affecting normals. Our key contribution lies in bridging classical graphics techniques, such as mesh-based deformation, with modern Gaussian primitives, achieving state-of-the-art geometry reconstruction and rendering quality. Unlike previous avatar rendering approaches, SurFhead enables efficient reconstruction driven by Gaussian primitives while preserving high-fidelity geometry.
BibTeX
@inproceedings{lee2025surfhead,
  author = {Jaeseong Lee and Taewoong Kang and Marcel C. Bühler and Min-Jung Kim and Sungwon Hwang and Junha Hyung and Hyojin Jang and Jaegul Choo},
  title = {{SurFhead: Affine Rig Blending for Geometrically Accurate 2D Gaussian Surfel Head Avatars}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=1x1gGg49jr},
}
Figure