Introduction
Traditional methods for editing visual content—such as adding, deleting, or moving objects—rely heavily on manual inputs like mice, touchscreens, or specialized controllers. In Augmented Reality (AR) environments, these modalities can be cumbersome and often present significant accessibility barriers for users with physical disabilities. GazeGen introduces a hands-free alternative, allowing users to generate and edit visual content using only their eye gaze. By integrating an ultra-lightweight gaze-tracking model with generative AI, GazeGen enables intuitive, real-time digital environment manipulation on edge devices.
The DFT Gaze Agent
The core of the GazeGen system is the DFT Gaze (Distilled and Fine-Tuned Gaze) agent. To operate effectively on wearable edge devices like the Raspberry Pi, the model must be both accurate and computationally efficient.
Compact Distillation and Personalization
The researchers achieved a model size of just 281K parameters through self-supervised compact distillation. Knowledge was transferred from a larger teacher model (ConvNeXt V2-A) using a Masked Autoencoder (MAE) approach, where the student model learns to reconstruct internal features from masked eye images.
To account for individual anatomical differences in eye shape and movement, GazeGen utilizes "Adapters"—small, tunable modules. This allows the system to be personalized for a specific user with as few as five eye images, significantly improving accuracy without the need for full model retraining.
Gaze-Driven Content Generation
Once the gaze point is accurately predicted, it serves as an active user interface element. The system defines a "Region of Interest" (ROI) around the gaze point, which is then processed by various generative models.
The pipeline supports several interaction modes:
- Detection: Using LLaVA or YOLOv9 to identify objects and suggest bounding boxes at the gaze location.
- Editing: Leveraging Stable Diffusion for adding, deleting, or changing the style of objects within the ROI.
- Animation: Utilizing Text-to-Video (T2V) models to animate static regions that the user is currently observing.
By focusing analysis only on the feature grid cells surrounding the gaze point, GazeGen drastically reduces the computational overhead typically associated with full-frame high-resolution processing.
Experimental Results
The system was validated using the OpenEDS2020 and Aria Everyday Activities (AEA) datasets. Key findings include:
- Efficiency: The DFT Gaze model is approximately 10x smaller than its teacher model while maintaining a mean angular error between 2.14° and 2.60°.
- Edge Performance: On a Raspberry Pi 4, the model achieved a latency of 426.66 ms, twice as fast as the teacher model.
- Versatility: Qualitative tests demonstrated complex gaze-triggered tasks, such as replacing a kitchen backsplash with an aquarium or animating a static river in a photograph.
Discussion and Future Work
While GazeGen demonstrates the potential for gaze-based content creation, several challenges remain. Environmental factors like glare can interfere with gaze estimation, and the system currently lacks a mechanism to handle blinks. Furthermore, ensuring that generated content perfectly matches the 3D perspective and lighting of the original scene is an ongoing area of research. Future iterations aim to integrate 3D modeling to improve the "grounding" of generated objects within physical spaces.
Summary updated