When you look at someone's eyes during a standard video call, you are looking at your display screen, not the camera lens. Because the camera is typically mounted above or to the side of the screen, your gaze appears directed downward or away from the perspective of the remote participant. This geometric discrepancy is known as camera-display parallax, and it fundamentally breaks mutual eye contact.
A nearly identical problem occurs in augmented and virtual reality headsets featuring "video passthrough." The external cameras capturing the physical world are located centimeters away from the user's actual pupils. This spatial offset alters depth perception and disrupts spatial awareness, often leading to disorientation and cybersickness.
Across telepresence and mixed reality, a major research direction focuses on resolving this viewpoint mismatch. By combining computer vision, machine learning, and novel hardware configurations, researchers aim to dynamically realign the captured image with the user's true line of sight.
The Geometry of Broken Eye Contact
To understand why digital communication can feel impersonal, consider the angular offset between a display and a camera. As noted by Antonio Criminisi et al. et al. in Gaze Manipulation for One-to-one Teleconferencing, this physical displacement prevents the simultaneous transmission and reception of a direct gaze. When one participant looks at the other's face on screen, they appear to be looking away, disrupting the natural dynamics of face-to-face social coordination (Kübra Bodur et al. in Using video calls to study children's conversational development: The case of backchannel signaling).
This issue scales poorly in multi-party conversations. In a traditional video conferencing grid, the flat representation lacks spatial awareness; it is impossible to perceive who is looking at whom (Yizhong Zhang et al. in VirtualCube: An Immersive 3D Video Communication System). Furthermore, standard wide-angle lenses often capture a distant, impersonal view that diminishes visual intimacy and engagement (Ross Cutler et al. in Multimodal active speaker detection and virtual cinematography for video conferencing). As summarized by Gene Cheung et al. in Introduction to the Issue on Interactive Media Processing for Immersive Communication, overcoming this physical offset is essential for establishing natural social interaction in immersive communication.
Manipulating the Gaze Vector at the Pixel Level
One way to restore eye contact without changing hardware is to synthetically alter the eye region within the 2D video stream. The goal is to redirect the gaze vector by roughly 10 to 15 degrees so that the eyes appear to look directly into the lens.
The authors of Learning to look up: Realtime monocular gaze correction using machine learning (Daniil Kononenko et al.) achieved this in real time using a machine learning framework called Eye Flow Forests. Instead of generating entirely new pixels, the system learns a mapping to "copy-paste" and warp existing pixels from within the eye region, shifting the iris and pupil upward while preserving the surrounding facial structure.
To improve photorealism, Yaroslav Ganin et al. developed DeepWarp in DeepWarp: Photorealistic Image Resynthesis for Gaze Manipulation. They utilized a coarse-to-fine deep learning architecture combined with a specialized lightness correction module. This approach ensures that fine details—such as specular highlights and the precise geometry of the eyelids—remain consistent after the gaze vector is redirected, preventing the "uncanny valley" effect common in early generative methods.
Synthesizing Virtual Perspectives and Reprojecting Scenes
While local eye-warping corrects the gaze direction, it does not fix the perspective of the rest of the face. If a camera is mounted high above a monitor, the viewer still sees the top of the speaker's head. True alignment requires synthesizing an entirely new virtual camera viewpoint.
In Gaze Manipulation for One-to-one Teleconferencing, Antonio Criminisi et al. used a stereo pair of cameras and a three-plane dynamic programming algorithm to compute dense depth maps. By establishing stereo correspondence, they could algorithmically synthesize a virtual camera view from any position on the monitor, effectively placing a virtual lens directly behind the interlocutor’s eyes.
Alternatively, depth sensors can be paired with explicit 3D modeling. The authors of Real-Time Face Model Reconstruction and Synthesis Using an RGB-D Camera for Face Reorientation (Toshihiko Yamasaki et al.) used an RGB-D camera to reconstruct a 2.5D mesh of the user's face in real time. By mathematically re-projecting this 3D structure to face the viewer, they corrected both the gaze and the overall facial perspective at high frame rates.
Viewpoint Correction in Spatial Passthrough and AR
The challenge of viewpoint alignment becomes even more acute when moving from flat screens to head-mounted displays (HMDs) and spatial environments. In video see-through AR, displaying an uncorrected camera feed causes a severe perspective mismatch because the cameras are physically separated from the user's optical axis.
To mitigate the resulting depth distortion and cybersickness, Trishia El Chemaly et al. proposed Geometry Aware Passthrough (GAP) in Mind the GAP: Geometry Aware Passthrough Mitigates Cybersickness. GAP uses real-time depth estimation to reproject the captured camera images directly onto the user's actual optical axis. Even though this reprojection can introduce minor visual warping artifacts, aligning the perspective with the user's physical eyes significantly reduces nausea and disorientation.
A similar geometric correction is required for see-through displays that attempt to create a "transparent wall" effect. In On the Performance and Implementation of Parallax Free Video See-Through Displays, Ricardo Augusto Borsoi et al. demonstrated that combining user head tracking with a planar scene model can successfully compensate for the spatial separation between the background-capturing camera and the user's eyes, preserving the illusion of transparency.
When these spatial systems fail to account for perspective, the illusion of presence collapses. For instance, Weizhan Zhang et al. showed in Real-and-Present: Investigating the Use of Life-Size 2D Video Avatars in HMD-Based AR Teleconferencing that representing remote participants as flat 2D planes in a 3D space fails when an observer moves laterally, because the flat representation cannot provide the correct parallax or perspective changes. Furthermore, large face-to-face collaborative displays introduce unique cognitive loads, such as orientation reversal (left/right confusion), which require software-based awareness widgets to resolve (Jiangtao Gong et al. in Side-by-Side vs Face-to-Face: Evaluating Colocated Collaboration via a Transparent Wall-sized Display).
Hardware Integration and Interaction Design Alternatives
Rather than relying entirely on post-capture image processing, some systems address the parallax problem through hardware integration or interface constraints.
As surveyed by Xiaohan Fang et al. in Immersive video interaction system: a survey, one direct hardware solution is to embed hidden camera arrays directly behind high-resolution, semi-transparent displays. This physically aligns the camera's perspective with the user's line of sight, eliminating the gaze offset at the point of capture.
Traditional Setup: Hidden Camera Array:
[Camera] [ Display ]
| \ [Hidden Camera]
| \ Gaze Vector |
[Display] | Gaze & Capture Aligned
Alternatively, the physical behavior of users can be guided through clever interaction design. In The effect of field of view on social interaction in mobile robotic telepresence systems, Amy Loutfi et al. discovered that narrowing a telepresence robot's camera field of view to a portrait orientation naturally encourages remote users to better orient the robot toward the local user. By limiting the horizontal view, the interface inherently promotes a direct, vis-a-vis interaction that minimizes look-away behaviors.
Finally, if the physical parallax cannot be eliminated, it can be explicitly compensated for in the user interface. The authors of Restoring Eye Contact to the Virtual Classroom with Machine Learning (Ross Greer et al.) developed a real-time machine learning pipeline that estimates which participant a user is looking at despite the camera-monitor parallax. Instead of warping the video feed, the system overlays the target's name on screen, explicitly communicating the user's intent and restoring the non-verbal cues necessary for effective remote collaboration.
Key Foundational Work
Gaze Manipulation for One-to-one Teleconferencing (2003) — contributed 1 concept elements to this direction.
Joint gaze-correction and beautification of DIBR-synthesized human face via dual sparse coding (2014) — contributed 1 concept elements to this direction.
The effect of field of view on social interaction in mobile robotic telepresence systems (2014) — contributed 1 concept elements to this direction.