Fooling the Geometric Eye: Physical Attacks on Gaze Estimators with a Refractive Gaussian-Splat Renderer

Computers increasingly infer where a person is looking — for virtual-reality displays, driver monitoring, and attention-aware interfaces — and they do so in two very different ways. Appearance-based models learn a statistical mapping from face pixels to gaze direction; geometric models instead fit the eye's anatomy: the iris is a circle in 3D that projects as an ellipse, and infrared "glints" — specular reflections off the curved cornea — can be traced back to locate the eye's optical axis. The appearance-based family is known to be hackable in the digital domain: small perturbations steer their outputs almost arbitrarily. But nobody has shown whether these attacks survive the trip into the physical world, and nobody has seriously attacked the geometric family at all. Meanwhile, the survey evidence says physical adversarial attacks via differentiable rendering have stayed almost entirely on discrete classification and vehicle detection — continuous regression tasks like gaze, depth, and motion are nearly untouched (RenderBender found depth and motion misestimation in just one paper each, and gaze in none).

This proposal argues that the geometric family is not robust by construction, and that breaking it requires — and motivates — a new piece of infrastructure: a fast differentiable renderer of the human eye that includes how the cornea and the fluid behind it bend light. Accurately simulating that refraction today means iterative ray tracing through layered media, which is exactly what has kept refractive eye models out of real-time and optimization pipelines (Price et al. had to precompute a look-up table offline for this reason). Our plan is to bake refraction into a Gaussian-splatting eye model so the adversarial optimization loop runs at explicit rasterization speeds, then use that renderer to build and physically test wearable attacks, map the failure modes of both estimator families, and train defenses that close the gap.

Core Claim

Three specific claims. First, geometric gaze estimators resist the texture-patch attacks that break CNN-based ones — because a patch on a cheek hijacks a CNN's attention but is invisible to a method that only fits the limbus ellipse and glint positions — yet they are vulnerable to a complementary physical channel: refractive eyewear and printed geometry near the eye that displace the apparent limbus and glints. The two families of estimators should therefore have largely disjoint physical failure sets, which no one has measured. Second, an anatomically faithful differentiable eye renderer is achievable at interactive speeds by baking ray-traced refraction into the Gaussian representation, making expectation-over-transformations attack loops tractable. Third, adversarial training driven by this physically faithful renderer should beat digital-perturbation adversarial training, which Xu et al.. showed yields only modest gains (attack deviation fell from about 60.6° to about 50.7°).

Research Program

The renderer. We represent the eye region as 3D Gaussians — fuzzy colored ellipsoids in space that a GPU rasterizes — attached to a morphable face mesh, following the attachment strategy of FlashAvatar and the two-stream separation of ocular and facial Gaussians in GazeGaussian, which rotates eye Gaussians explicitly when gaze changes. The new ingredient is refraction. Offline, we run a one-time anatomical ray trace in the spirit of Price et al. — cornea ($n \approx 1.376$), aqueous humor ($n \approx 1.336$), aspheric anterior surface — over a grid of gaze angles, corneal surface points, and eyewear lens configurations, and fit a compact differentiable warp that maps each eye-region Gaussian to its apparent refracted image position as a function of viewing geometry: the Gaussian analog of their 3D look-up table. At runtime the renderer is pure rasterization, avoiding both the neural-renderer overhead that caps GazeGaussian at 74 FPS and the dense MLP queries that make NeRF-style compositors like GazeNeRF unusable inside an optimization loop. This is the same trade-memory-for-speed move as SNeRG, applied to optical physics rather than scene radiance. We validate the warp against fresh ray-traced ground truth on glint position and limbus-ellipse geometry, and check that it reproduces the Price phenomenon: oblique-view iris texture should match head-on enrollments far better after an inverse-warp correction, consistent with the 27.4% median Hamming-distance reduction (70.0% at 20°–25° tilts) they report. No reported result found on Lacuna for the throughput of iterative refractive eye ray tracing, so we measure both paths ourselves.

The attack. The attacker controls eyewear $\theta$: a printable pattern on the frame and an optional thin transparent refractive element in front of the eye, drawn from a small set of fabricable lens surfaces. The optimization maximizes the angular error between the victim's predicted gaze and an attacker-chosen target, averaged over a nuisance distribution $v$ of head pose, true gaze, illumination, camera geometry, and subject anatomy (using the head–eye offset coupling of Wang et al. so head motion and eye rotation stay anatomically consistent):

$$\theta^{*} = \arg\max_{\theta}\; \mathbb{E}_{v}\Big[\, \angle\big(G(\mathcal{R}_{\text{refr}}(\theta; v)),\; g^{\text{tgt}}\big) \Big],$$

where $\mathcal{R}_{\text{refr}}$ is our baked-refraction renderer, $G$ the victim estimator, and $g^{\text{tgt}}$ the attacker's desired gaze vector. Printability and wearability enter as constraints: total-variation smoothness and a non-printability score for the pattern (the recipe of Pautov et al., including their grayscale trick for printer–camera gamut shifts), and a dioptric-power cap on the lens so the wearer's own vision is essentially unimpaired. This is expectation over transformations — optimizing the average effect across a distribution of conditions rather than one fixed view — with the same machinery the physical-attack literature uses (Zhou et al. for weather and lighting; REVAMP shows how refraction itself can enter the loop, but at simulation cost). Victims span both families: appearance models (Full-Face and Gaze-Net, as evaluated by Xu et al.) and geometric models — single-image limbus fitting via the one-circle algorithm of Wang et al., glint-based unprojection following Hosp et al. and Perra. We then physically realize the attacks: printed patterns and procured lens elements mounted on eyeglass frames, worn by consenting subjects, captured across webcams, phone cameras, and a near-infrared rig with active illumination. All human-subject work proceeds under institutional review and with the defensive framing stated up front.

The robustness study and the defense. The scientific payoff is a cross-victim attack-success matrix under physical capture: we expect texture patches to move appearance models but not geometric ones, and refractive elements to move geometric ones but not appearance models. If confirmed, this argues for hybrid estimators. We then retrain victims with the surrogate renderer in the loop — augmenting training with physically rendered attacks rather than digital noise — and compare against the digital adversarial-plus-stability training that Xu et al. found only weakly effective. A secondary application closes the loop with latency research: foveated VR pipelines like A3FR steer high-resolution rendering from a live gaze estimate, so an attack that biases gaze by a few degrees misplaces the foveal region entirely; we quantify that downstream effect on rendering quality.

Baselines

The naive reference point is the victim estimators under benign conditions: clean images, ordinary glasses, and randomly textured (non-optimized) frames. No reported result found on Lacuna for the clean angular error of Full-Face or Gaze-Net on these benchmarks, so we establish these numbers ourselves; an attack counts only if it induces error far beyond the spread of these benign-wearable controls. The strongest existing competitor is the only published attack aimed at gaze victims: the digital patch attack of Xu et al., which achieved an angular error to the attacker's target of 10.13° on Full-Face (their pixel-level iterative attack reached 0.60°, the digital ceiling). We physically realize their patch using the print pipeline of Pautov et al. — grayscale optimization, momentum-iterative gradients with total-variation loss, homography-based placement — because their machinery is the strongest demonstrated physical eyewear attack, achieving a cosine similarity of 0.363 to the target identity (versus 0.305 for the wearer's true identity) in face verification. A second strong baseline ports that Pautov pipeline directly onto our gaze victims, optimized without any eye model. The key ablation runs our full pipeline with refraction disabled — a standard Gaussian compositor in the GazeGaussian style — to isolate what the refractive ingredient buys. Fairness is enforced rigidly: identical victim checkpoints, the same expectation-over-transformations iteration count and nuisance distributions, the same print and capture hardware and subject pool, and the same hyperparameter search budget (learning rates, smoothness and printability weights) for every attack method. Beating the baselines means a targeted physical attack whose error-to-target under real captures is at or below 10.13° — matching the best reported digital figure under harder physical conditions — while clearly exceeding both the physically realized Xu patch and every random-texture control, and, for the defense, cutting physical attack deviation by more than the roughly 16% improvement that digital adversarial training delivered in Xu et al..

Why not just…?

Why not just print the existing digital patches and test them physically? This is the real threat to the proposal's novelty, and we run it as the primary baseline rather than arguing around it. Two pieces of evidence say it falls short. Xu et al. themselves leave the physical-to-digital gap open and their Score-CAM analysis shows the patches work by hijacking CNN attention to irrelevant facial regions — a mechanism a geometric estimator, which never looks at the cheeks, simply does not have. And RenderBender documents that high-frequency digital patterns routinely die in the print-and-capture pipeline, which is why physical attacks need macro-structure and plausibility constraints. But honesty requires noting that a patch printed near the eye could still bias limbus localization in geometric victims; if the baseline matches our attack on all victims, the simpler route wins and we will report that as the finding.

Why not just use an off-the-shelf differentiable 3D Gaussian renderer, as physical camouflage attacks already do? PGA demonstrated that Gaussian splatting works well as the differentiable bridge for vehicle camouflage, and TT3D did the analogous thing with NeRF features. But those pipelines contain no eye: they cannot represent how eyewear bends light before it reaches the cornea, nor how the cornea warps apparent geometry at oblique angles — precisely the physics our attack channel exploits and the physics geometric victims depend on. The ablation baseline settles this empirically: if the refraction-free compositor transfers to physical captures at oblique head poses as well as the refractive one, the ingredient is unnecessary. We expect the gap to grow with viewing angle, consistent with the angle-dependent distortion Price et al. measured up to 60° tilts.

Why not just adversarially fine-tune the gaze model and skip the renderer entirely? The simplest defense route, and Xu et al. already tried a version of it: adversarial plus stability training reduced attack deviation only from about 60.6° to about 50.7°, and their perturbations were digital, so the training distribution never covered print degradation, lens optics, or real lighting. Our defense experiment compares both under the same tuning budget; if digital adversarial training closes most of the physical gap anyway, that too would be a worth-reporting negative result — but the pattern in the physical-attack literature (RenderBender) is that distribution mismatch, not gradient masking, is what defeats defenses.

Decisive Checks

Four outcomes decide the proposal. The renderer must match ray-traced ground truth on glint and limbus geometry within a tight pixel tolerance, reproduce the Price distortion phenomenon qualitatively, and render eye-region crops at hundreds of frames per second, so that a full expectation-over-transformations optimization converges within a day of GPU time — against which a GazeNeRF-style compositor or iterative ray tracing should be orders of magnitude slower, a comparison we will report as wall-clock numbers. The complementarity matrix is the decisive scientific result: a clean split in which texture patches move appearance victims only and refractive elements move geometric victims only, established on the benchmarks the field already uses (MPIIFaceGaze, ETH-XGaze, ColumbiaGaze, and EYEDIAP, following Xu et al. and GazeNeRF, which reports 6.94° gaze angular error on ETH-XGaze redirection as the fidelity reference our renderer must not degrade). The physical attack must reach an error-to-target at or below the 10.13° digital baseline across real capture conditions, beating the physically realized Xu patch. And the renderer-in-the-loop defense must cut physical attack deviation by more than the roughly 16% that digital adversarial training achieved. If all four hold, the field gains its first physically validated attack on a continuous regression task with anatomical modeling, a map of which estimator designs are worth deploying where gaze is safety-critical, and a reusable fast refractive eye renderer for both adversarial training and foveated VR systems like A3FR.