In a video call, nobody can tell who is looking at whom. In a meeting room, a glance tells a speaker whether the listener is following, who is about to talk, and who has dropped out. On a laptop, the same glance lands on a small rectangle of pixels, one tile in a grid, and the cue is lost. This note proposes to recover that cue as a measurement. For each participant, a model would estimate which tile on that participant's own screen they are looking at, moment to moment, using only the webcam feed and the layout the conferencing client already knows. The proposal does not need a per-user calibration routine or labeled call footage. It trains a small "attention decoder" entirely on simulated meetings and then adapts it on real unlabeled calls. The adaptation uses a self-supervised signal that video calls provide for free: when the client rearranges tiles, a person who keeps watching the same colleague must move their eyes by exactly the known on-screen displacement.
The problem is hard, and it has not been solved by simply running a gaze estimator, for a geometric reason. On a 13-inch laptop viewed from about 55 cm, the centers of adjacent tiles in a 3×3 gallery sit roughly 10° apart horizontally and only about 6° apart vertically. This follows from simple trigonometry on a ~29×18 cm screen. The best cross-domain gaze estimators in this research file still err by 5.7° to 9.3° (Generalizing Gaze Estimation with Rotation Consistency; PureGaze). Much of that error is a fixed per-person offset. Gaze error is therefore the same size as tile spacing. Turning a gaze angle into "Alice is looking at Bob" is a separate inference problem, and that inference is where simulation and self-supervision can help.
Core Claim
The claim is that tile-level visual focus of attention (VFOA: the identity of the target a person is looking at) can be estimated in real multi-party video calls without per-user calibration. The estimate would match or beat a calibrated gaze pipeline. It needs two ingredients. The first is a decoder trained on simulated sessions that randomize screen geometry, camera placement, layouts, gaze-estimator noise, and conversational attention behavior. The second is unsupervised adaptation on real calls driven by layout-reshuffle consistency. This consistency is the video-call counterpart of the rotation-consistency trick: in that trick, rotating a face rotates its gaze by a known angle. Here, moving a tile moves the attended point by a known vector.
Simulation is used differently here than in the closest prior work. Multistream Gaze Estimation With Anatomical Eye Region Isolation trains an eye-segmentation network purely on UnityEyes renders with heavy noise augmentation and shows that it transfers to real eyes. Its output is a single person's gaze angle from an eye crop. Existing meeting-attention systems operate in physical rooms with calibrated sensors and real annotations: VFOAmap uses Kinect-based head and gaze maps, and ICAF models interdependent gaze on AMI. None of them faces the distributed setting. There, each person's "room" is a private screen with its own layout, and the targets are tiles rather than 3D bodies. This proposal does not render photorealistic faces. Appearance is the layer where the sim-to-real gap is worst, and PixelDA-style translation cannot handle structural differences (PixelDA). The simulation covers only the layers that are cheap to model faithfully: geometry, layout dynamics, sensor-level gaze noise, and turn-taking-conditioned attention behavior.
Research Program
The system has three parts. The first is a frozen appearance backbone: a gaze and head-pose estimator trained on real data (ETH-XGaze source) and generalized with an existing method such as RUDA or PureGaze. The proposal does not try to improve this stage. It outputs a noisy gaze ray and head pose per frame.
The second part is the attention decoder. It is a small temporal model that takes three inputs: the gaze/head-pose stream, the client's layout log (tile rectangles and the identity in each), and an audio-derived conversational state (who is speaking, recent turn changes). It outputs a distribution over targets: each tile, the shared-screen region, the user's own self-view, and "off-screen." The decoder is trained only on simulated sessions. The simulator samples screen size, viewing distance, and camera offset from the screen. It also samples grid layouts and speaker-view switches, and reorder events at realistic rates. It corrupts the true gaze ray with an error model fitted to the residuals the backbone produces on MPIIGaze (MPIIGaze). That error model includes a persistent per-person bias, frame-level jitter, and failures under low light and extreme pose.
Attention behavior comes from a randomized policy. Listeners favor the active speaker, glance at addressees and at people who backchannel, look away at a randomized aversion rate, and look at shared content. The policy's parameters are drawn from wide ranges rather than fixed. The aversion range brackets the 17% and 39% rates reported for KTH-Idiap and UBImpressed. This is domain randomization applied to behavior rather than pixels. The aim is for the decoder to learn to integrate geometry and context without memorizing one behavioral style.
The third part is adaptation on real unlabeled calls. Let $g_t$ be the decoder's estimated on-screen gaze point at time $t$, and let $p_t(k)$ be the known screen position of target $k$'s tile. Across a reshuffle between times $t_1$ and $t_2$, the adaptation loss is
$$ L_{\text{shuffle}} = \sum_k w_k \,\big\| (g_{t_2} - g_{t_1}) - (p_{t_2}(k) - p_{t_1}(k)) \big\|^2 , $$
where $w_k$ is the decoder's posterior probability that the user attends target $k$ both before and after the change. Because the loss uses differences, any fixed per-user offset cancels. That is why it can stand in for calibration. A light anchoring term then sets the absolute offset. It pulls the slow-moving per-user bias estimate toward positions consistent with speaker-attention events.
This loss has a known failure mode: the model can fit its own wrong pseudo-labels. Error amplification of this kind is documented for regression adaptation (PnP-GA). The proposal adopts that paper's remedy of a momentum teacher providing the posteriors $w_k$, and it restricts the loss to reshuffles where the gaze jump is unambiguous. The data collection client will also inject mild, controlled tile swaps (roughly one per minute) so that reshuffle events are plentiful rather than incidental.
The data required is new, because the file contains no tile-level video-call attention benchmark. Groups of four will hold 30-minute video calls on two tasks. One is a remote pair-programming variant with two observers, chosen to match the setting of Detecting Exclusive Language during Pair Programming. The other is a hidden-profile decision task in which knowledge is unevenly distributed, echoing Desert Herding. Each participant uses their own laptop webcam at home. An instrumented client logs layouts. A screen-mounted eye tracker, used only for ground truth, provides tile labels. Roughly 40 groups would give about 80 participant-hours. Participants will be split by person and by group for all evaluations.
Baselines
The naive reference is the active-speaker heuristic: each listener is predicted to look at the current or most recent speaker's tile, and the speaker is predicted to look at the last person who spoke. It needs no camera at all. It is a meaningful floor because ICAF and VFOAmap both treat speaking status as a top cue for attention. Any vision-based method that does not clearly beat it is not reading gaze. It will be scored on the same frames, and beating it means a statistically reliable gain in frame-level tile accuracy and in macro-F1 over targets. Macro-F1 matters because this heuristic is strong on "speaker" frames and blind to aversion and side-glances.
The strongest uncalibrated competitor maps a generalized gaze estimator through assumed geometry. The ETH-XGaze-trained backbone is adapted with RUDA, which reports 5.70° on ETH-XGaze→MPIIFaceGaze (Rotation Consistency), or with PureGaze, which reports 9.32° on Gaze360→EyeDiap (PureGaze). Its gaze ray is intersected with a screen plane at default laptop geometry and assigned to the nearest tile. A variant also uses PnP-GA's unsupervised adaptation on our unlabeled call footage, with the same backbone and the same unlabeled data budget as our adaptation stage. For tile-level accuracy in video calls I found no reported result on Lacuna, so this baseline will be measured here. The closest reported figure is VFOAmap's cross-dataset accuracy of up to 85% in co-located meetings with calibrated RGB-D sensors. That figure is not directly comparable, but it indicates what "good" looks like in an easier sensing regime.
The strongest practical competitor is the same pipeline plus a per-user 9-point explicit calibration at the start of each call. The calibration fits a per-user offset and scale. I treat it as the bar the proposal must reach. Beating it is not required for the claim to hold. The claim is matching it within two points of tile accuracy without any calibration, and beating it after posture drift, measured in the last third of each session.
Ablations isolate the two key ingredients. The first ablation keeps the decoder architecture but trains it on a small labeled real subset instead of simulation. The second trains on simulation with a fixed behavior policy, or with uniformly random attention, to test whether behavioral randomization matters. The third removes $L_{\text{shuffle}}$ and keeps only speaker anchoring. The fourth keeps $L_{\text{shuffle}}$ but drops the injected swaps, relying only on organic reshuffles. Every variant gets the same backbone, the same unlabeled hours, and the same hyperparameter search budget, with search done on held-out groups.
Why not just…?
Why not just calibrate each user and use a good gaze estimator? This is the real threat, and the proposal is built so that it can lose to it visibly. Calibration removes the per-person bias that dominates cross-domain gaze error. On a fixed posture it may well match the decoder. The case against it is practical and temporal. Participants rarely tolerate calibration in every meeting. Laptop posture shifts during a call. The decisive check is accuracy as a function of minutes since calibration. If calibrated accuracy holds flat over 30 minutes and stays above the adapted decoder, the contribution shrinks to "calibration-free at a stated cost," and I will report it that way.
Why not just use the active speaker as the attention target? The speaker cue is strong but structurally incomplete. Listeners look at reactors, shared content, and away. The aversion rates of 17% and 39% in the VFOAmap datasets show that a large share of frames is not speaker-directed even face to face. The heuristic is also useless for the speaker's own gaze, which is exactly the signal that indicates whom a turn is addressed to. The macro-F1 comparison above, broken down by target class, will show whether vision adds anything beyond this cue.
Why not just fine-tune the gaze network on real video-call footage? That requires tile-level labels. Producing them needs either eye-tracker hardware or slow frame-by-frame annotation, the same bottleneck that keeps labeled social-signal data small, as the exclusive-language work illustrates with 40,490 utterances yielding only 0.45 F1 on the rare class. The "real labeled subset" ablation measures how many labeled participant-hours are needed to match the simulation-trained decoder. If the answer is "two hours," simulation is not worth it.
Why not just apply generic feature-level domain adaptation? Feature-alignment methods such as DANN and DSN were far behind structured approaches on synthetic-to-real pose estimation: 56.58° and 53.27° versus PixelDA's 23.5° (PixelDA). Rotation-consistent adaptation likewise beat adversarial gaze adaptation (Rotation Consistency). The bias here is also a per-user offset in screen coordinates, not a shift in the feature distribution. A difference-based constraint cancels such an offset. Distribution matching does not target it at all. The PnP-GA baseline variant covers this family empirically.
Proposed Experiments
The primary experiment compares tile accuracy and target macro-F1 on held-out groups across the naive, uncalibrated, calibrated, and proposed pipelines. Results are stratified by layout density (2×2 versus 3×3), lighting, and minutes into the session.
The first decisive check is whether simulation-only training already beats the uncalibrated geometric baseline before any real adaptation. If it does not, the simulated geometry and noise model are wrong in a way that the ablations should localize.
The second decisive check is whether $L_{\text{shuffle}}$ closes the gap to the calibrated pipeline. Its effect must hold when swaps are organic rather than injected, or deployment would require modifying the client's behavior.
A behavioral-transfer check scores the simulation-trained decoder's conversational component on the co-located KTH-Idiap and AMI recordings with geometry inputs replaced by true target positions. Comparing this against VFOAmap and ICAF's reported accuracy tells us whether the randomized behavior policy captures real attention dynamics, separately from the geometry.
A downstream check tests whether the recovered cue matters for coordination signals. Attention features are added to the audio-only turn-taking judge from Talking Turns, which reports about 92% ROC-AUC on turn states, and we measure whether they improve prediction of turn changes and addressee on our calls. Separately, we test whether attention graphs, unlike the speech-only signals in Meeting Mediator, relate to task outcomes on the hidden-profile task. That study changed turn-taking but not task performance, so a null here would be informative rather than surprising.
References
- Generalizing Gaze Estimation with Rotation Consistency
- PureGaze: Purifying Gaze Feature for Generalizable Gaze Estimation
- Multistream Gaze Estimation With Anatomical Eye Region Isolation by Synthetic to Real Transfer Learning
- Visual Focus of Attention Estimation in 3D Scene With an Arbitrary Number of Targets
- Predicting the Visual Focus of Attention in Multi-Person Discussion Videos
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
- MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation.
- Generalizing Gaze Estimation with Outlier-guided Collaborative Adaptation
- Detecting Exclusive Language during Pair Programming
- Conversation dynamics in a multiplayer video game with knowledge asymmetry
- Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics
- Meeting mediator: enhancing group collaborationusing sociometric feedback