GazeGen: Gaze-Driven User Interaction for Visual Content Generation

He-Yen Hsieh, Ziyun Li, Sai Qian Zhang, Wei-Te Mark Ting, Kao-Den Chang, Barbara De Salvo, Chiao Liu, H. T. KungarXiv 2024 · DOI · arXiv

Introduction

Traditional methods for editing visual content—such as adding, deleting, or moving objects—rely heavily on manual inputs like mice, touchscreens, or specialized controllers. In Augmented Reality (AR) environments, these modalities can be cumbersome and often present significant accessibility barriers for users with physical disabilities. GazeGen introduces a hands-free alternative, allowing users to generate and edit visual content using only their eye gaze. By integrating an ultra-lightweight gaze-tracking model with generative AI, GazeGen enables intuitive, real-time digital environment manipulation on edge devices.

The DFT Gaze Agent

The core of the GazeGen system is the DFT Gaze (Distilled and Fine-Tuned Gaze) agent. To operate effectively on wearable edge devices like the Raspberry Pi, the model must be both accurate and computationally efficient.

The GazeGen pipeline begins with capturing the user's view and estimating gaze in real-time using the DFT Gaze Agent.

Compact Distillation and Personalization

The researchers achieved a model size of just 281K parameters through self-supervised compact distillation. Knowledge was transferred from a larger teacher model (ConvNeXt V2-A) using a Masked Autoencoder (MAE) approach, where the student model learns to reconstruct internal features from masked eye images.

To account for individual anatomical differences in eye shape and movement, GazeGen utilizes "Adapters"—small, tunable modules. This allows the system to be personalized for a specific user with as few as five eye images, significantly improving accuracy without the need for full model retraining.

Gaze-Driven Content Generation

Once the gaze point is accurately predicted, it serves as an active user interface element. The system defines a "Region of Interest" (ROI) around the gaze point, which is then processed by various generative models.

Gaze-driven interaction enables tasks such as object detection, image editing (e.g., inserting a teddy bear), and region-specific animation.

The pipeline supports several interaction modes:

  • Detection: Using LLaVA or YOLOv9 to identify objects and suggest bounding boxes at the gaze location.
  • Editing: Leveraging Stable Diffusion for adding, deleting, or changing the style of objects within the ROI.
  • Animation: Utilizing Text-to-Video (T2V) models to animate static regions that the user is currently observing.

By focusing analysis only on the feature grid cells surrounding the gaze point, GazeGen drastically reduces the computational overhead typically associated with full-frame high-resolution processing.

Experimental Results

The system was validated using the OpenEDS2020 and Aria Everyday Activities (AEA) datasets. Key findings include:

  • Efficiency: The DFT Gaze model is approximately 10x smaller than its teacher model while maintaining a mean angular error between 2.14° and 2.60°.
  • Edge Performance: On a Raspberry Pi 4, the model achieved a latency of 426.66 ms, twice as fast as the teacher model.
  • Versatility: Qualitative tests demonstrated complex gaze-triggered tasks, such as replacing a kitchen backsplash with an aquarium or animating a static river in a photograph.

Discussion and Future Work

While GazeGen demonstrates the potential for gaze-based content creation, several challenges remain. Environmental factors like glare can interfere with gaze estimation, and the system currently lacks a mechanism to handle blinks. Furthermore, ensuring that generated content perfectly matches the 3D perspective and lighting of the original scene is an ongoing area of research. Future iterations aim to integrate 3D modeling to improve the "grounding" of generated objects within physical spaces.

Summary updated

Abstract

We present GazeGen, a user interaction system that generates visual content (images and videos) for locations indicated by the user's eye gaze. GazeGen allows intuitive manipulation of visual content by targeting regions of interest with gaze. Using advanced techniques in object detection and generative AI, GazeGen performs gaze-controlled image adding/deleting, repositioning, and surface style changes of image objects, and converts static images into videos. Central to GazeGen is the DFT Gaze (Distilled and Fine-Tuned Gaze) agent, an ultra-lightweight model with only 281K parameters, performing accurate real-time gaze predictions tailored to individual users' eyes on small edge devices. GazeGen is the first system to combine visual content generation with real-time gaze estimation, made possible exclusively by DFT Gaze. This real-time gaze estimation enables various visual content generation tasks, all controlled by the user's gaze. The input for DFT Gaze is the user's eye images, while the inputs for visual content generation are the user's view and the predicted gaze point from DFT Gaze. To achieve efficient gaze predictions, we derive the small model from a large model (10x larger) via novel knowledge distillation and personal adaptation techniques. We integrate knowledge distillation with a masked autoencoder, developing a compact yet powerful gaze estimation model. This model is further fine-tuned with Adapters, enabling highly accurate and personalized gaze predictions with minimal user input. DFT Gaze ensures low-latency and precise gaze tracking, supporting a wide range of gaze-driven tasks. We validate the performance of DFT Gaze on AEA and OpenEDS2020 benchmarks, demonstrating low angular gaze error and low latency on the edge device (Raspberry Pi 4). Furthermore, we describe applications of GazeGen, illustrating its versatility and effectiveness in various usage scenarios.
BibTeX
@misc{hsieh2024gazegen,
  author = {He-Yen Hsieh and Ziyun Li and Sai Qian Zhang and Wei-Te Mark Ting and Kao-Den Chang and Barbara De Salvo and Chiao Liu and H. T. Kung},
  title = {{GazeGen: Gaze-Driven User Interaction for Visual Content Generation}},
  year = {2024},
  eprint = {2411.04335},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.04335},
}
Figure