Correcting Gaze Mismatch in Digital Communication

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 215 papers23 conceptsfrozen 2026-10-04

In a natural conversation, eye contact is the bedrock of social presence. It signals engagement, builds trust, and facilitates non-verbal cues. However, in video conferencing, this connection is fundamentally "broken" by a simple geometric reality: the camera is almost never where the other person's eyes are.

When you look at your colleague’s eyes on your screen, you are looking away from your camera. To your colleague, you appear to be looking down. If you look directly into the camera to appear engaged, you can no longer see your colleague’s facial expressions. This physical displacement between the lens and the display creates a "gaze gap" that Gene Cheung and colleagues identify as a primary barrier to immersive communication in Introduction to the Issue on Interactive Media Processing for Immersive Communication.

The Geometry of the Gaze Gap

The problem is one of parallax. As Antonio Criminisi and Andrew Blake explain in Gaze Manipulation for One-to-one Teleconferencing, the goal is to synthesize a "virtual camera" view from the center of the monitor.

Early solutions attempted to solve this with hardware. As surveyed by Xiaohan Fang and Shiqi Wang in Immersive video interaction system: a survey, some systems use semi-transparent mirrors or hidden camera arrays positioned directly behind high-resolution displays. While effective, these setups are bulky and expensive. Research has shifted toward software-based "gaze redirection" that can work on standard laptops and smartphones.

Redirecting Gaze via Image Warping

The most intuitive software approach is to treat the eye as a 2D region that can be "warped" or "shifted." Daniil Kononenko and Victor S. Lempitsky pioneered this with Eye Flow Forests in Learning to look up: Realtime monocular gaze correction using machine learning.

Instead of trying to understand the 3D structure of the eye, their system learns a mapping: it "copy-pastes" pixels from within the eye region to shift the iris and pupil upward by 10–15 degrees. This was later refined into DeepWarp, where Yaroslav Ganin used deep learning to ensure that fine details, like the specular highlights (the "glint" in the eye), remain photorealistic even after the eye is moved.

3D Reconstruction and Neural Synthesis

While warping works for small corrections, it struggles with larger head rotations. To address this, researchers like Toshihiko Yamasaki and Kiyoharu Aizawa utilized RGB-D cameras (which capture depth) to build a 2.5D model of the face. In Real-Time Face Model Reconstruction and Synthesis Using an RGB-D Camera for Face Reorientation, they demonstrate that by re-projecting this 3D model, they can virtually "tilt" the entire face toward the viewer, maintaining natural eye contact even if the user is looking significantly away from the lens.

More recently, generative models have moved away from explicit geometry. Jichao Zhang and Hao Tang proposed GazeCorrection, which treats gaze redirection as an image inpainting problem. Using Generative Adversarial Networks (GANs), the system "erases" the original eyes and synthesizes new ones looking at the camera, allowing for gaze correction in "in-the-wild" photos where depth data isn't available.

The Sensitivity of the Human Observer

The difficulty of this task stems from the fact that humans are evolutionarily tuned to detect gaze. As Yuang Zhang notes in ShoulderShot: Generating Over-the-Shoulder Dialogue Videos, even a minor misalignment—just a few degrees—can make a conversation partner appear "checked out" or emotionally disconnected.

This sensitivity becomes even more acute in Augmented Reality (AR). Weizhan Zhang and Christian Sandor found that when remote participants are represented as 2D video avatars in a 3D space, the illusion of presence breaks the moment the viewer moves. Without proper parallax and gaze correction, the avatar looks like a "cardboard cutout" rather than a person standing in the room.

By bridging the gap between where we look and where we appear to look, this research direction aims to transform video calls from a broadcast of a face into a genuine, spatially-consistent social encounter.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 17 papers
  2. Version 1 · 2026-10-04 · 15 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_parallax_induced_gaze_misalignment_0f533193,
  title = {parallax-induced gaze misalignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parallax-induced-gaze-misalignment/rdr_0f533193f65d481b80108443ac40be5e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_correcting_gaze_mismatch_in_digital_comm_0f533193_v1,
  title = {Correcting Gaze Mismatch in Digital Communication},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/correcting-gaze-mismatch-in-digital-communication/rdr_0f533193f65d481b80108443ac40be5e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}