TL;DR
This paper introduces EgoNav, a novel egocentric human trajectory prediction system for wearable robots. It leverages a diffusion model and a new "visual memory" representation to predict future paths, outperforming baselines in collision avoidance and capturing diverse movement patterns. A new large-scale egocentric walking dataset is also released to advance research in this area.
What Problem Does This Paper Address?
Predicting human motion is crucial for collaborative robots and wearable assistive devices, especially those worn by humans (like exoskeletons). These robots need to understand their wearer's egocentric perspective to navigate complex environments safely and effectively. Traditional methods often struggle with the multi-modal nature of human movement (multiple possible paths) and the challenges of egocentric vision (occlusions, limited field of view). Existing datasets often use a bird's-eye-view (BEV) perspective, which simplifies the problem but doesn't reflect the reality of egocentric sensing. This paper aims to bridge this gap by developing a robust egocentric trajectory prediction system that accounts for scene context and can generate diverse, plausible future paths.
Methods
The core of EgoNav is a diffusion model adapted for trajectory prediction. This model takes the wearer's past trajectory and a novel "visual memory" representation as input. The visual memory is a compact, panoramic representation of the surrounding environment, constructed by fusing depth, RGB, and semantic segmentation data from the wearer's egocentric sensors. This representation aims to capture both appearance and semantic information of the scene.
To make real-time inference feasible, the authors introduce a hybrid generation technique. This method combines the speed of DDIM sampling with the quality of DDPM sampling, allowing for rapid generation of multiple trajectory samples. The diffusion model itself uses a UNet architecture with self-attention layers to better relate visual memory to trajectory predictions.
Datasets
The paper introduces the Egocentric Navigation Dataset, a new, large-scale dataset collected from human participants wearing egocentric sensors (Intel Realsense T265 and D455). It captures diverse indoor and outdoor walking scenarios with detailed state and visual information, including RGB, depth, and semantic segmentation. The dataset comprises 198 minutes of data (over 400GB) and is intended to facilitate research in egocentric trajectory prediction. The authors also mention curating a smaller, high-quality pilot dataset for faster iteration during development.
Key Insights and Ideas
- Visual Memory is Key: The "visual memory" representation, which stitches together egocentric sensor data into a panoramic view, is crucial for capturing relevant scene context that a single frame would miss. This is particularly important for unstructured walking environments.
- Diffusion Models for Multi-Modality: Diffusion models are well-suited for generating diverse, multi-modal trajectory predictions, which is essential for capturing the inherent uncertainty and multiple viable paths in human navigation.
- Hybrid Generation for Real-Time: The hybrid DDIM/DDPM sampling approach is a critical innovation for making diffusion-based trajectory prediction practical for real-time applications, balancing speed and quality.
- Egocentric Data is Underrepresented: The creation of a large-scale, egocentric dataset highlights the need for more data that reflects the perspective of wearable robots and assistive devices.
Key Results
- EgoNav outperforms baseline methods (LSTM-VAE, CXA Transformer) on key metrics, particularly in collision avoidance (achieving the highest collision-free score) and trajectory mode coverage (indicated by a good "Best of N" score).
- The inclusion of semantic information in the visual memory significantly improves collision avoidance.
- The hybrid generation technique is 33 times faster than a full DDPM baseline, achieving a good balance between speed and prediction quality.
- The model can generate a distribution of 15 different trajectory samples in real-time.
Limitations and Future Work
- Stereo Depth Dependency: The current method relies on stereo depth cameras. Future work could explore using monocular depth estimation models to relax this requirement.
- Computational Cost: While the hybrid generation improves speed, diffusion models are still computationally intensive. Distilling the latent diffusion model for even greater efficiency is suggested.
- Markovian Assumption: The paper explores a non-Markovian assumption by using a long history of past trajectories. Further research could investigate reducing this history for potential computational gains, though it might impact smoothness and mode coverage.
- Interaction Modeling: The current work focuses on predicting trajectories without explicit interaction modeling with other agents. Future work could incorporate these interactions.
Summary updated