TL;DR
This paper presents an image-based human action recognition system designed to assist visually impaired individuals. It utilizes an ear-mounted camera and a smartphone to capture and transmit scenes to a server for analysis, which then provides auditory feedback on detected human actions via text-to-speech. The system achieves a 60.7% action recognition performance in real-world indoor and outdoor environments.
What Problem Does This Paper Address?
Visually impaired individuals face significant challenges in navigating and understanding their surroundings. While existing assistive technologies often rely on sensors like ultrasound or GPS, these methods can be limited by range, installation complexity, and the inability to provide comprehensive scene understanding. This paper addresses the need for a more sophisticated assistive system that can interpret dynamic human actions within a scene, going beyond simple object detection. The authors highlight the gap in research concerning the recognition of human actions in conjunction with object recognition for scene understanding, particularly for the visually impaired.
Methods
The proposed system integrates several key components:
- Ear-mounted Bluetooth Camera: This wearable device allows the user to capture images of their surroundings.
- Smartphone Integration: The captured images are transmitted from the camera to a synchronized smartphone.
- Action Recognition Server: The smartphone relays the images to a server for processing. The server performs a sequential pipeline:
- Human Detection: The system employs a "part-based" approach using a sub-window boosting technique. This method focuses on learning optimal sub-regions of the human body rather than relying on global features, making it more robust to occlusions and pose variations. It utilizes a Random Forest classifier for learning.
- Object Detection: Similar to human detection, object detection also uses the sub-window boosting technique. A novel aspect here is the introduction of a "shared part" strategy to handle multi-view object recognition efficiently. This approach identifies overlapping sub-regions across different viewpoints to reduce computational load.
- Human Action Recognition: This module utilizes "poselets," which are defined as collections of body parts with similar attributes. The system trains a two-level Random Forest classifier. The first level uses poselet classifiers for each action, and the second level combines these results to make a final action classification.
- Text-to-Speech (TTS) Output: The recognized action information is sent back to the smartphone, which then converts it into audible speech for the user.
Datasets
- Human Detection: INRIA dataset was used for training and evaluation.
- Object Detection: PASCAL VOC 2012 dataset was used.
- Human Action Recognition: PASCAL 2010 and PASCAL 2011 datasets were utilized.
The paper also mentions experimental results using data captured by the proposed system in indoor and outdoor environments.
Key Insights and Ideas
- Part-based Detection for Robustness: Focusing on specific body parts for human and object detection proves more effective than global features, especially in cluttered or occluded scenes.
- Shared Parts for Efficient Multi-View Object Recognition: The strategy of identifying and sharing common sub-regions across different object viewpoints significantly improves computational efficiency for multi-view object recognition.
- Poselets as Action Primitives: Decomposing human actions into minimal units (poselets) allows for effective action recognition even from static images, by capturing the essential pose information.
- Two-Level Action Recognition: The hierarchical approach to action recognition, first classifying poselets and then combining these results, offers a structured way to handle complex actions.
- Integrated System for Real-World Application: The successful integration of these individual modules into a complete system, from capture to TTS output, demonstrates a practical approach to assistive technology.
Key Results
- Human Detection: The proposed method achieved a precision of 89% and a recall of 87% on the INRIA dataset, showing improvements over the Histogram of Oriented Gradients (HoG) method.
- Object Detection: The system achieved an average precision of 48.8% and a recall of 57.8% on the PASCAL VOC 2012 dataset, with a slight improvement in precision over the Selective Search method.
- Human Action Recognition: The system demonstrated an average precision of 60.7% on the PASCAL datasets. While the accuracy was only slightly better than a related study, it achieved a significantly faster recognition speed (3.81 seconds faster).
- Overall System Performance: In real-world experiments using the developed system, an action recognition performance of 60.7% was achieved on test data captured in indoor and outdoor environments. The overall system processing time, including capture, transmission, and TTS, averaged around 20 seconds, with action recognition being the most time-consuming part.
Limitations and Future Work
- Processing Time: The current system's action recognition module is computationally intensive, leading to a significant processing delay. The authors acknowledge that increasing the number of action classes further exacerbates this issue.
- Network Latency: The transmission of images between the smartphone and the server introduces delays, especially with higher resolution images.
- Specific Action Recognition Accuracy: Certain actions, like "phoning" and "taking a picture," showed lower recognition rates (24% and 22% respectively) due to their similarity to other actions.
- Future Work:
- Contextual Information: Incorporating background and situational context, in addition to object-human interactions, is suggested to improve recognition accuracy for ambiguous actions.
- System Optimization: Reducing computation time and network transmission delays are key areas for future improvement.
- Integration with Wearable Devices: Exploring integration with devices like Google Glass for enhanced service creation.
- Expanding Action Classes: Increasing the variety of recognizable human actions.
- Robustness to Diverse Poses and Objects: Improving the recognition of human poses and objects in a wider range of scenarios.
Summary updated