How Much of Cross-Dataset Gaze Error Is in the Labels?

Gaze estimation networks predict where a person is looking from a photo of their face. When such a network is tested on a dataset other than the one it was trained on, its error roughly doubles (Cheng et al.'s review and benchmark). The field treats this as a vision problem: lighting, cameras and faces differ, so the network must learn features that ignore them. Part of the gap, however, may not be the network's fault. Every gaze dataset records "ground truth" through its own hardware rig and calibration. Every person's line of sight also differs from the visible axis of their eyeball by a personal angle, called angle kappa, that cannot be seen in any image. A test label can therefore disagree with what the face shows by a fixed, person-specific rotation that no appearance model could ever predict.

We propose to measure that unpredictable part directly, for the standard cross-domain test sets MPIIFaceGaze and EyeDiap, with no designated "gold standard" dataset. The core move is borrowed from image-classification label audits. If many independently trained models, built on different architectures and trained on data from different capture rigs, all make the same per-person error in the same direction, that shared error most likely lives in the test labels, or in the eye anatomy those labels encode, rather than in any one model. We turn this agreement signal into per-subject label-frame corrections, a corrected evaluation protocol, and a test of whether the recent cross-domain gaze literature has hit a noise ceiling that hides real differences between methods.

Core Claim

Our claim is that a substantial, measurable fraction of the cross-domain error on MPIIFaceGaze and EyeDiap is a per-subject rigid rotation that is shared across diverse models. If so, this shared rotation sets a floor on reported error and compresses the apparent gaps between methods. The NLP audits found the same effect: cleaning TACRED raised average F1 by 14.3% and separated C-GCN from PA-LSTM, which had looked identical on the noisy data (Re-TACRED). The file gives circumstantial support for such a floor in gaze. On ETH-XGaze, fine-tuning on 200 samples from one person cuts ResNet-50 error from 4.5° to about 2° (ETH-XGaze). Nine calibration samples bring MPIIGaze to 3.14° (FAZE). Meanwhile, uncalibrated cross-domain methods cluster between roughly 5.5° and 6.6° on MPIIFaceGaze (CRGA reports 5.48°; GLA reports 6.58°). Nobody has separated how much of that cluster reflects label frame rather than model quality.

Relation to Prior Work

Four papers sit at this intersection, and each constrains or motivates the design. Gaze Label Alignment (GLA) shows in simulation that rig miscalibration produces up to 3° of systematic label deviation. It corrects for this by fitting a rotation and translation that maps each dataset's labels onto a manually chosen anchor dataset. This motivates our premise, but GLA inherits whatever errors the anchor carries, and its reported gain is modest (6.83° to 6.58°). We use no anchor. We fix the overall coordinate frame by averaging across rigs, and we identify label error by agreement across models rather than by agreement with one reference dataset.

QualitEye checks whether eye poses that look the same map to the same labels across datasets, using hashed, privacy-preserving codes. Its hash buckets are coarser than the 1–3° offsets we target, it also depends on a reference, and it flags samples without correcting them. GLSGE handles shift in the overall label distribution: wide gaze ranges in ETH-XGaze versus narrow ones in MPIIFaceGaze. It explicitly assumes the recorded labels are correct, which is the assumption we test. EIF documents that mixing datasets hurts source accuracy and absorbs the conflicting labels as irreducible measurement noise. We instead ask whether that "noise" has structure, specifically a per-person rotation, that can be estimated and removed.

The feature-level generalization methods supply the diverse model pool our identification argument needs, but they are not competitors on the auditing question. These include PureGaze, De-confounded Gaze Estimation, RUDA, 3DPE-Gaze and SG-Gaze. SG-Gaze is a useful constraint: it openly ignores angle kappa and loses within-domain accuracy as a result. This is consistent with kappa being invisible to appearance, and therefore something to measure rather than learn.

Research Program

We first build a deliberately diverse pool of roughly 30 gaze models. They vary along three axes: architecture (ResNet-18/50, GazeTR, PureGaze, FSCI using the official code), training source (ETH-XGaze, Gaze360, GazeCapture, and their union), and random seed. All are trained with GazeHub normalization so that preprocessing is identical.

Each target subject's test frames are split into a fitting half and an evaluation half, stratified by gaze direction. On the fitting half we estimate, for every model $m$ and subject $s$, the 3D rotation $R_{m,s}$ that best maps the model's predictions onto the labels. We then factor it as

$$R_{m,s} \approx L_s \, B_{m,s},$$

where $L_s$ is a label-frame rotation shared by all models for subject $s$, and $B_{m,s}$ is that model's own bias. We estimate $L_s$ by robust rotation averaging, which finds the rotation closest to all the per-model rotations, while penalizing $B_{m,s}$ toward identity.

The identification argument rests on diversity. Any offset a model inherits from its source rig differs between ETH-XGaze, Gaze360 and GazeCapture models, so it ends up in $B_{m,s}$. An offset that persists across all three source rigs and across architectures must come from the target labels, or from anatomy that appearance does not reveal. For our purposes these two are equivalent: either way, no appearance model can be expected to remove the error.

We then report three numbers per method on the evaluation half. The first is raw angular error. The second is error after applying only the shared $L_s$, which is a label-frame-corrected score that does not depend on the method being scored. The third is the oracle per-model calibrated error. The difference between raw and $L_s$-corrected error is our estimate of the label floor.

A second stage tests whether the rigid model is enough. It replaces $L_s$ with a smooth function of gaze direction, fitted with a low-order spherical-harmonic warp, to check whether lens and screen-projection distortions add a gaze-dependent component, as the file's gap analysis suggests for EyeDiap and Gaze360 rigs. A third stage applies the same consensus procedure to the source datasets, with leave-one-subject-out models within each source. We retrain on the corrected source labels as an anchor-free alternative to GLA. We will release the corrections as per-subject rotation patches, which sidesteps redistribution licensing in the way Re-TACRED did.

Baselines

The naive reference point is the standard protocol: mean angular error on uncorrected MPIIFaceGaze and EyeDiap labels, computed on exactly the same evaluation halves. For orientation, the file reports a 6.83° baseline on MPIIFaceGaze (GLA), 8.33° for ResNet-18 on the combined MPIIFaceGaze+EyeDiap targets (GLSGE), and 11.42° from Gaze360 to EyeDiap (PureGaze). Our estimate counts as meaningful only if the shared correction lowers error well beyond what bootstrap resampling of subjects would produce by chance.

The strongest existing label-correction method is GLA, which reports 6.83° to 6.58° on MPIIFaceGaze (GLA). We will rerun GLA with each of ETH-XGaze, Gaze360 and GazeCapture as the anchor, using the same backbone, training schedule and hyperparameter search budget as our retraining stage. Beating it means two things. Anchor-free retraining must reach lower target error than the best GLA anchor. Our corrections must also vary less across the choice of source than GLA's results vary across the choice of anchor. Since no anchor-sensitivity numbers for GLA appear on Lacuna, the second comparison is itself new.

The strongest method that uses target labels is per-model, per-subject calibration in the spirit of FAZE, which reports 3.14° with 9 samples on MPIIGaze. This is an oracle, not a competitor. It bounds how much any rotation-based correction can achieve, and the fraction of its gain that is shared across models is exactly what we report.

For methods that treat conflicting labels as noise, we include EIF at its reported 6.23° average on unseen MPIIFaceGaze+EyeDiap. We ask whether its predicted uncertainty is higher for subjects with large estimated $|L_s|$; if it is not, EIF's uncertainty is missing the structured component.

The key ablation removes diversity. It builds the pool from a single architecture and source with different seeds. If the "shared" offset barely changes, then diversity is not doing the identifying work, and we would be measuring the shared bias of one source rig instead of label error.

Why not just…?

Why not just report error after few-shot personal calibration, which already removes per-person offsets? This is the closest threat, and our protocol contains it as the oracle. The problem with using it as the metric is that per-model calibration also erases genuine model failures that happen to be consistent within a person, such as failure on a particular eye shape. That would reward weak models. The shared-component estimate removes only offsets that every model makes. The comparison between oracle and shared correction will show how much of calibration's benefit is label frame and how much is model-specific. If the two coincide, the simple answer is right, and we will say so.

Why not just use GLA with a carefully chosen anchor? GLA's own gain is 0.25° (GLA), and its correctness depends on the anchor being accurate, which is exactly the assumption benchmark audits warn against. Our anchor-sweep experiment measures directly how much GLA's corrections move when the anchor changes. Large movement would mean anchor-based alignment imports the anchor's errors.

Why not just train bigger models on more mixed data? EIF shows naive mixing hurts source accuracy because labels conflict. More importantly, angle kappa is invisible in images, so no amount of capacity can predict it. If our shared offsets shrink as model size grows, that would falsify the claim that they are unpredictable from appearance, so model scale is one axis of the pool.

Why not just model label shift as GLSGE does? GLSGE reweights by the overall label distribution and assumes the labels are correct. A rigid rotation per subject leaves the marginal label range nearly unchanged, so reweighting cannot detect it. We will apply our audit to GLSGE-trained models to check whether their residuals still contain the shared offset.

Proposed Experiments

The first decisive check is synthetic recovery. We take held-out ETH-XGaze subjects, inject known per-subject rotations of 1–4°, which matches GLA's simulated range, and require the consensus estimate to recover them to within a fraction of a degree. We also confirm that injecting nothing yields offsets near zero.

The second check is the size of the floor on the real targets. If the shared correction removes less than roughly a fifth of cross-domain error on both MPIIFaceGaze and EyeDiap, the label-noise hypothesis is largely wrong for these benchmarks, and that negative result is worth publishing. EyeDiap offers an internal control: its screen-target and floating-ball sessions have different label geometry, so offsets should differ between session types for the same person if they come from the rig, and match if they come from anatomy.

The third check is architectural compression. We compute Kendall rank correlation between method rankings under raw and corrected scores, and bootstrap confidence intervals over the 15 MPIIFaceGaze and 16 EyeDiap subjects for every pairwise gap. The prediction is that gaps such as PureGaze versus ResNet, or 3DPE-Gaze versus ResNet, widen after correction, and that some currently reported orderings fall within the noise.

The fourth check is downstream. Retraining on consensus-corrected source labels should beat the best GLA anchor on raw target error, using matched compute.