Abstract
Egocentric vision holds great promises for increasing access to visual information and improving the quality of life for people with visual impairments, with object recognition being one of the daily challenges for this population. While we strive to improve recognition performance, it remains difficult to identify which object is of interest to the user; the object may not even be included in the frame due to challenges in camera aiming without visual feedback. Also, gaze information, commonly used to infer the area of interest in egocentric vision, is often not dependable. However, blind users often tend to include their hand either interacting with the object that they wish to recognize or simply placing it in proximity for better camera aiming. We propose localization models that leverage the presence of the hand as the contextual information for priming the center area of the object of interest. In our approach, hand segmentation is fed to either the entire localization network or its last convolutional layers. Using egocentric datasets from sighted and blind individuals, we show that the hand-priming achieves higher precision than other approaches, such as fine-tuning, multi-class, and multi-task learning, which also encode hand-object interactions in localization.
Introduction
For individuals with visual impairments, using a camera to identify objects presents a fundamental "chicken and egg" problem: to recognize an object, the camera must be aimed correctly, but without visual feedback, aiming is notoriously difficult. This often leads to "blind photography," where the target object is off-center, partially occluded, or entirely absent from the frame.
While sighted users' gaze can often indicate intent in egocentric video, this signal is unreliable for blind users. However, researchers have observed a consistent compensatory behavior: blind users frequently place their hands near or on the object they wish to identify to help guide the camera's field of view. This behavior suggests that the hand can serve as a powerful semantic anchor and spatial pointer.
Methodology: Hand-Primed Localization
The proposed approach introduces "hand-priming," a technique that uses hand segmentation as a contextual cue to locate objects of interest. The architecture is based on the FCN-8s model and consists of two primary components: a Hand Segmentation Network and an Object Localization Network.
The core innovation lies in the "priming" mechanism, where the hand mask is infused into the localization network. Two specific architectures were evaluated:
- HPAll (Hand-Primed All): The hand mask is integrated into every convolutional layer (conv1–conv5).
- HPLate (Hand-Primed Late): The hand mask is infused only into the later layers (conv4 and conv5).
These models were compared against standard baselines, including a "NoHand" model and a "Finetune" model (the previous state-of-the-art), as well as multi-task models that attempt to learn hand and object features simultaneously.
Datasets and Evaluation
The models were trained and tested on three egocentric datasets: GTEA, GTEA Gaze+ (kitchen tasks by sighted individuals), and TEgO (Teachable Egocentric Objects). TEgO is particularly significant as it contains images captured by a blind individual using a wearable camera to recognize personal items. To isolate the effect of hand-priming, the evaluation on TEgO focused specifically on images where a hand was present.
Results and Discussion
The HPLate model emerged as the superior architecture, consistently outperforming other methods across all metrics. By infusing hand information into the deeper layers of the network, the model effectively combines high-level descriptive features with the spatial context provided by the hand.
Key findings include:
- Significant Precision Gains: HPLate improved Average Precision (AP) by 17% on GTEA, 50% on GTEA Gaze+, and 12% on the TEgO dataset compared to the fine-tuned baseline.
- Reduction of False Positives: Hand-priming acts as a spatial filter, suppressing background clutter and "hallucinations" by focusing the model's attention on the area of interaction.
- Robustness Across Populations: The model achieved comparable performance on data from both sighted (AP 0.59) and blind (AP 0.57) users, validating the "hand-as-pointer" strategy for real-world assistive applications.
Conclusion
This research demonstrates that the hand, often treated as noise in computer vision, is a vital source of intent for assistive egocentric systems. By leveraging hand-object interactions, localization models can become significantly more accurate and reliable for users with visual impairments. Future work will focus on refining localization from coarse "center areas" to precise instance segmentations and addressing challenges in highly cluttered environments with multiple hands or objects.
Summary updated