TL;DR
This paper introduces ECC-Net, a deep learning model that corrects eye contact in video conferencing by redirecting gaze to the camera without requiring explicit gaze direction or camera/display geometry. It achieves this by learning to predict and manipulate gaze through a novel bi-directional training approach and a set of control mechanisms for natural, real-time performance.
What Problem Does This Paper Address?
Maintaining eye contact during video calls is crucial for effective communication, but the typical setup of having cameras and displays misaligned makes it challenging. Users naturally tend to look at the screen where participants appear, breaking the illusion of direct eye contact. Existing solutions often rely on specialized hardware or make strong assumptions about camera placement, which are not suitable for the diverse range of modern devices (smartphones, tablets) with cameras in various positions. This paper tackles the problem of achieving natural-looking eye contact correction in real-time, adaptable to different device orientations and user viewing distances, without requiring explicit user input about their gaze.
Methods
The core of the proposed solution is ECC-Net, a deep convolutional neural network. It takes an image patch of a single eye as input and outputs a vector field and a brightness map.
- Vector Field: This field guides the warping of the input image to redirect the eye's gaze.
- Brightness Map: This map is used to adjust local brightness, enhancing eye clarity and correcting artifacts introduced by the warping process, particularly in the eye whites.
A key innovation is the bi-directional training approach. The model is trained to both correct the gaze (from input image to corrected output) and reconstruct the original image from the corrected output. This cycle-consistent loss, inspired by image-to-image translation techniques, helps to reduce artifacts and enforce mapping reversibility.
Additionally, the paper introduces control mechanisms to ensure a natural user experience. These mechanisms dynamically disable correction based on factors like face size, distance, head pose, eye opening, and blinking, preventing "creepy" or unnatural results when the user is not engaged in a typical video conferencing scenario.
Datasets
The authors utilized two datasets:
- Natural Dataset: Collected from over 200 participants, consisting of 3125 gaze pair sequences where subjects looked at both the camera and random points on a display. This dataset was primarily used for validation and refining the synthetic data.
- Synthetic Dataset: Generated using the UnityEyes platform and refined by a generative adversarial network (GAN). This dataset provided a large volume of photorealistic, perfectly labeled data, crucial for training the deep learning model. The GAN refinement process aimed to make the synthetic data distribution closer to real-world data.
The Columbia Gaze Dataset was used as a public benchmark for testing, re-organized to match the paper's pair-wise gaze format.
Key Insights and Ideas
- Implicit Gaze Prediction: A surprising finding is that the model implicitly learns to predict the input gaze direction as a byproduct of the eye contact correction task. This is attributed to the need to infer gaze when no explicit redirection angle is provided.
- Bi-directional Training for Artifact Reduction: The cycle-consistent loss in bi-directional training is crucial for mitigating warping artifacts and ensuring that the corrected gaze looks natural, rather than relying solely on a single-direction correction.
- Control Mechanisms for Naturalness: The inclusion of control mechanisms that dynamically enable/disable correction based on user behavior is vital for a smooth and non-intrusive user experience, preventing corrections in non-standard scenarios.
- Synthetic Data with GAN Refinement: Generating a large, perfectly labeled synthetic dataset and then refining it with a GAN to match real-world distributions offers a powerful way to train robust models without the cost and complexity of extensive real-world data collection.
Key Results
- ECC-Net achieves a significant reduction in mean squared error compared to identity transform (59% for the model trained on refined synthetic data and 56% for the baseline DeepWarp model).
- The model trained solely on refined synthetic data achieved a validation error of 0.375 and a test error of 0.414, outperforming the baseline DeepWarp model (0.411 validation, 0.442 test) under similar training conditions.
- The control mechanisms significantly improve performance by filtering out problematic frames, reducing validation error from 0.423 to 0.375 and test error from 0.442 to 0.414.
- The system runs in real-time on a commodity CPU, making it feasible for a wide range of devices.
- Qualitative results show that the corrected images preserve details like glasses and eyelashes, avoiding hallucinated artifacts.
Limitations and Future Work
- Limitations: The paper acknowledges that in extreme cases, such as nearly closed eyes, the model might need to be smoothly disabled to avoid unnatural corrections. The gaze prediction capability, while useful, is not as accurate as dedicated gaze prediction models and does not incorporate head pose or distance from the screen.
- Future Work: The authors suggest that the gaze prediction capability could enable hands-free interactions (e.g., dimming backlights) and that the system could be applied to other areas like software teleprompters and personal broadcasting applications. Further exploration of the implicit gaze prediction could lead to more advanced functionalities.
Summary updated