TL;DR
This paper presents a novel, efficient, and robust deep learning-based system for pedestrian detection and human-aware navigation (HAN) for robots. By cascading a fast feature detector (ACF) with a deep CNN, they achieve real-time pedestrian detection. This is then integrated with standardized human-aware navigation constraints to enable robots to interact safely and naturally with people in shared environments.
What Problem Does This Paper Address?
Navigating in environments shared with humans presents a significant challenge for robots. To interact naturally and safely, robots need to not only detect and track people but also understand and adhere to social norms and personal space requirements. This is known as Human-Aware Navigation (HAN). Existing approaches often rely solely on onboard sensors, limiting their perception, or use computationally expensive methods for pedestrian detection, hindering real-time performance. This paper aims to bridge this gap by proposing an efficient deep learning-based pedestrian detection system that can be integrated into a robot's navigation stack, allowing it to dynamically adjust its path based on human presence and social rules.
Methods
The core of the proposed system lies in a two-stage approach for Pedestrian Detection (PD):
- Cascaded Detector: They combine the Aggregate Channel Features (ACF) detector with a deep Convolutional Neural Network (CNN). The ACF acts as a fast, selective proposal generator, identifying regions of interest likely to contain pedestrians. These proposals are then fed into a CNN for more accurate classification. This cascade significantly reduces the computational load on the CNN, enabling real-time performance.
- Deep Learning for PD: A pre-trained VGG-VD16 CNN model, initially trained on ImageNet, is fine-tuned for pedestrian detection using the INRIA dataset. To achieve real-time performance on a CPU, the input image size for the CNN is reduced from 224x224 to 64x64.
- Human-Aware Navigation Constraints: The paper standardizes and reformulates several HAN constraints, including personal space (modeled using asymmetric and circular Gaussians depending on the person's posture), visibility, interaction, and social rules (like overtaking on the left). These constraints are integrated into a costmap used by an A* path planner.
- People Tracking: A Nearest Neighbor Joint Probabilistic Data Association (NNJPDA) algorithm with Kalman filters is used to track detected pedestrians in the world coordinate system, providing velocity estimates crucial for HAN constraints.
Datasets
- Pedestrian Detection Training/Evaluation:
- INRIA Dataset: Used for training and evaluating the pedestrian detection module. It consists of 1832 training images and 288 test images.
- Realistic Scenarios:
- "Corridor" dataset: 5556 images.
- "MBOT" dataset: 3966 images. These datasets were acquired using a combination of onboard and fixed ceiling-mounted cameras.
Key Insights and Ideas
- Cascading for Efficiency: The key insight is that a fast, albeit less accurate, detector like ACF can significantly speed up a more powerful but slower detector like a CNN by acting as a selective pre-filter. This is crucial for real-time robotic applications.
- Standardization of HAN Constraints: By reformulating and standardizing HAN constraints, the authors aim to create a more unified and integrable framework for robot navigation in human environments.
- CPU-Based Real-Time Performance: The paper demonstrates that with careful model adaptation (e.g., input size reduction) and algorithmic design (cascading, thresholding), real-time pedestrian detection and navigation can be achieved even on CPU-based robotic platforms, making it more accessible.
- Importance of Multi-Camera Systems: The use of both onboard and external cameras provides a more comprehensive view of the environment, improving robustness and enabling detection even when a person is not in the onboard camera's field of view.
Key Results
- Pedestrian Detection Accuracy: The cascaded ACF+CNN approach achieved a log average miss rate of 15.13% on the INRIA dataset, an improvement over the ACF proposals alone (16.83%).
- Real-Time Performance:
- On the INRIA dataset, the ACF+CNN system achieved 6.62 FPS with thresholding.
- In real scenarios ("corridor" and "MBOT" datasets), the system achieved approximately 10 FPS with thresholding, which is considered suitable for real-time applications. This is a significant improvement over an exhaustive search CNN approach, which was estimated at 0.1238 FPS.
- Human-Aware Navigation: Simulated and realistic experiments demonstrated that the robot successfully navigated while respecting personal space, avoiding interference with human interactions, and adhering to social rules like overtaking on the left. The system showed robustness to occlusions and dynamic changes in the environment.
Limitations and Future Work
- Limitations:
- The paper acknowledges that the CNN model's input size was reduced for speed, which might slightly impact accuracy.
- While the system performed well with seated persons, it wasn't explicitly designed for all such cases.
- The fine-tuning of the CNN hyperparameters was not extensively optimized.
- Future Work:
- The authors suggest that the CNN could be further fine-tuned with a specific HAN dataset for even better performance.
- The paper mentions that the proposed approach is general and can be applied to other non-deep detectors.
- The authors plan to make their ROS packages for PD and HAN publicly available.
Summary updated