Abstract
Gaze redirection—modifying an image so a subject appears to look in a specific direction—is a complex task requiring the preservation of identity and 3D consistency. Traditional 2D-based methods often struggle with "identity bleed" or anatomical inaccuracies. We introduce GazeNeRF, a volumetric approach that leverages Neural Radiance Fields (NeRF) to model the face and eyeballs as distinct entities. By applying explicit 3D rotations to eye features within a two-stream architecture, GazeNeRF achieves state-of-the-art performance in gaze accuracy and head pose consistency while maintaining high-fidelity identity preservation.
Introduction
The human eye is a 3D sphere rotating within a socket, a physical reality that 2D image manipulation techniques, such as pixel warping or GANs, often fail to capture. These methods frequently result in artifacts where changing the gaze inadvertently alters the facial structure or fails to maintain consistency across different head poses. While some models attempt to incorporate 3D priors into latent spaces, they lack explicit physical modeling. GazeNeRF addresses these challenges by reformulating gaze redirection as a 3D-aware volumetric task, ensuring that eye movements are anatomically and geometrically plausible.
Methodology: The Two-Stream Architecture
The core of GazeNeRF is a novel two-stream MLP architecture designed to disentangle the eyeballs from the rest of the face.
- Two-Stream Modeling: The system utilizes two separate MLP streams: one for the "face-only" region (skin, head shape, sockets) and another specifically for the "eyeballs." This separation allows the model to learn independent representations for static and dynamic components.
- Explicit 3D Rotation: Because the eyes are isolated in their own feature volume, the model can apply a standard 3D rotation matrix ($R$) directly to the eye features. This explicitly mimics the physical rotation of an eyeball in 3D space.
- Volume Compositing and Rendering: The rotated eye features are merged with the static face features using a
maxoperation. This combined volume is then processed by a neural rendering module to produce high-resolution (512x512) output images. - Conditioning and Loss: The model is conditioned on 3D Morphable Model (3DMM) parameters to ensure identity consistency. Training involves a combination of reconstruction loss, perceptual loss, and a specialized functional loss guided by a pre-trained gaze estimator.
Experiments and Results
GazeNeRF was evaluated on several prominent datasets, including ETH-XGaze for primary training and MPIIFaceGaze, ColumbiaGaze, and GazeCapture for cross-dataset generalization.
Quantitative Performance
GazeNeRF significantly outperforms previous state-of-the-art methods like STED (2D-based) and HeadNeRF (general face NeRF). On the ETH-XGaze dataset, GazeNeRF reduced gaze error to 6.94° (down from STED's 16.22°) and improved head pose accuracy to 3.47° error (compared to STED's 13.15°). It also achieved higher Structural Similarity (SSIM) scores, indicating superior identity preservation.
Qualitative Comparison
Visual results demonstrate that GazeNeRF avoids common 2D artifacts such as "blurry eyes" or the "sliding face" effect. By modeling the eyes volumetrically, the system ensures they appear properly seated within the sockets regardless of the redirection angle.
Conclusion
GazeNeRF demonstrates that explicit 3D modeling is fundamentally superior to implicit 2D learning for tasks involving rigid rotations like eye movement. By disentangling the face and eyes into separate volumetric streams, the model achieves unprecedented accuracy and realism. While current limitations include high training costs and sub-real-time inference speeds (~1 second per image), GazeNeRF sets a new benchmark for 3D-aware facial manipulation and opens doors for high-quality synthetic data generation in gaze estimation tasks.
Summary updated