3D Geometric Eye Modeling

36 concepts24 papers1 proposal

When we observe a human eye in a standard 2D image, we are looking at a complex flat projection of a dynamic, three-dimensional structure. If a person shifts their gaze or turns their head, the circular shape of the iris warps into an ellipse, the pupil shifts relative to the eyelids, and the surrounding facial features distort under perspective projection.

Traditional appearance-based gaze tracking methods treat this as a black-box image mapping problem, requiring massive datasets to interpolate between every possible combination of eye appearance, head pose, and camera angle. 3D geometric eye modeling takes the opposite approach: it builds explicit anatomical and physical priors directly into the system. By treating the eye as a geometric entity—such as a sphere or an ellipsoid anchored within the 3D structure of the face—we can mathematically decode eye movements, eliminate tedious calibration, and synthesize highly realistic animations.

From 2D Pixels to 3D Spheres

The fundamental advantage of geometric eye modeling is its ability to handle perspective distortion mathematically rather than statistically. When the eyeball rotates, the circular iris projects onto a 2D image plane as an ellipse. By explicitly modeling the eyeball as a 3D sphere or a series of nested ellipsoids, the relationship between the 3D optical axis and its 2D image projection becomes deterministic (Salam et al.; Wang et al.).

   3D Eyeball Rotation              2D Image Projection
     _..---.._                         .-------.
   .'  .---.  '.                      /   ---   \  <- Iris appears
  /   /     \   \                    |   /   \   |    as an ellipse
 |   |   +   |   |  ============>>   |  |  +  |  |    at extreme
  \   \     /   /                    |   \   /   |    gaze angles
   '.  '---'  .'                      \   ---   /
     `''---''`                         '-------'

This structural constraint drastically reduces the need for dense training data. For instance, Funes Mora demonstrated that incorporating an explicit 3D eyeball model allows gaze estimation systems to accurately extrapolate gaze directions far outside the boundaries of the training data. While interpolation-based 2D models fail when encountering unseen gaze angles, a geometric model simply continues to rotate the underlying 3D sphere.

Solving the Head-Eye Offset

A major hurdle in remote gaze tracking is that the eyes do not move in isolation; they are rigidly embedded within a moving head. If a system treats facial tracking and eye tracking as separate pipelines, any head movement introduces severe errors into the estimated gaze vector. To translate 2D image features into a reliable 3D gaze vector, a system must know the exact spatial relationship between the surface of the face and the center of the eyeball—a metric known as the head-eye offset.

To bypass the need for manual calibration of this offset, Wang et al. integrated the eyeball center as a fixed geometric component directly inside a 3D deformable facial model. Because the spatial relationship between the facial landmarks and the eyeball center is structurally locked, the system can calculate the 3D gaze vector by finding the intersection between a line from the eyeball center and the detected pupil position. This unified modeling allows the gaze vector to remain stable even during significant, unconstrained head movements.

A similar principle applies to broader tracking tasks: Hu demonstrated that using a 3D cylinder as a geometric proxy for the human head simplifies six-degree-of-freedom motion tracking, allowing 2D image gradients from multiple camera perspectives to be mapped onto a shared, consistent 3D coordinate system.

Unprojecting Corneal Reflections and Refraction

To achieve high precision without specialized, expensive hardware, geometric eye modeling often leverages corneal reflections, or "glints," produced by infrared light sources. By mathematically unprojecting these reflections from the 2D image back into 3D space, systems can calculate the exact center of the cornea and derive the eye's optical axis (Hosp et al.; Perra). Even when restricted to a single high-resolution image of a single eye, the "one-circle" algorithm developed by Wang et al. leverages these geometric constraints and anatomical priors to resolve perspective projection ambiguities entirely.

However, the physical eye is not a perfect mirror; it is a complex optical system. Light traveling from the iris must pass through the aqueous humor and the cornea, bending nonlinearly due to refraction before it reaches the camera lens. This refraction distorts the visible iris pattern, particularly at off-axis viewing angles.

  Camera
    \
     \      Air
  -----\----------------- Cornea Surface
        \   Aqueous Humor
         \__
            \
             | Iris Pattern (Refracted Position)

Price et al. showed that by incorporating an anatomical ray-trace model that explicitly simulates these refractive components, systems can mathematically "undo" this geometric distortion. Correcting for these refractive effects allows iris recognition systems to maintain high accuracy even at extreme off-axis angles of up to 60 degrees.

Overcoming Parallax and Projection Distortions

When users interact with environments at varying depths, traditional 2D mapping methods suffer from parallax errors. By utilizing a physical 3D model of the eye alongside binocular cameras, systems can track the convergence of both gaze vectors in 3D space (Toivanen et al.). This allows the system to compute the exact gaze distance, ensuring accuracy whether the user focuses on a near or far object. Furthermore, the authors of Probabilistic gaze estimation without active personal calibration demonstrated that combining this 3D physical eye model with environmental saliency maps allows the system to incrementally learn a user's unique eye parameters through natural viewing, completely eliminating the need for active personal calibration.

Geometric modeling also proves vital when the visual environment itself is distorted, such as in 360-degree panoramic images. In a flat 2D equirectangular projection of a panorama, straight sight lines appear as complex curves, and the edges create artificial coordinate discontinuities. Shen et al. and Quan et al. resolved this by modeling gaze paths directly as 3D sight lines in a spherical coordinate system. Operating in 3D spherical space eliminates projection artifacts entirely, maintaining spatial consistency across the poles and longitudinal boundaries.

Furthermore, when these 3D gaze vectors are combined with explicit 3D scene geometry and occlusion reasoning, systems can resolve visual ambiguities so effectively that they match or exceed human-level performance in gaze target detection (Bao et al.). This spatial precision is critical for downstream applications, such as calculating the precise visual angle of objects to determine user attention in wearable AI systems (Wilson et al.).

Dense Geometry for Photorealistic Synthesis

The utility of 3D eye geometry extends beyond tracking and estimation into generative tasks like gaze redirection and facial reenactment. Traditional generative approaches manipulate eyes in 2D, which frequently causes unnatural stretching and identity loss because they ignore the physical rotation of the eyeball within its socket.

To fix this, Ruzzi et al. introduced GazeNeRF, which uses a neural radiance field to model the face and eyes as separate 3D volumes, applying explicit 3D rotations directly to the eye features before rendering. Similarly, the authors of Free-HeadGAN demonstrated that regressing full, dense 3D eye meshes—rather than simple directional vectors—captures the nuanced geometric changes of the eye structure across different viewing angles. These dense meshes provide precise structural guidance for image synthesis, enabling highly realistic gaze transfer while perfectly preserving the subject's identity.

Key Foundational Work

On the Efficacy of Correcting for Refractive Effects in Iris Recognition (2007) — contributed 1 concept elements to this direction.
Eye Gaze Estimation from a Single Image of One Eye (2003) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_geometric_eye_modeling_e19b1f70,
  title = {geometric eye modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/geometric-eye-modeling/rdr_e19b1f704cd44509b792e6eafe627093},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_geometric_eye_modeling_e19b1f70_v1,
  title = {geometric eye modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/geometric-eye-modeling/rdr_e19b1f704cd44509b792e6eafe627093/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake