Abstract
A multi-modal framework to generate user intention distributions when operating a mobile vehicle is proposed in this work. The model learns from past observed trajectories and leverages traversability information derived from the visual surroundings to produce a set of future trajectories, suitable to be directly embedded into a perception-action shared control strategy on a mobile agent, or as a safety layer to supervise the prudent operation of the vehicle. We base our solution on a conditional Generative Adversarial Network with Long-Short Term Memory cells to capture trajectory distributions conditioned on past trajectories, further fused with traversability probabilities derived from visual segmentation with a Convolutional Neural Network. The proposed data-driven framework results in a significant reduction in error of the predicted trajectories (versus the ground truth) from comparable strategies in the literature (e.g. Social-GAN) that fail to account for information other than the agent's past history. Experiments were conducted on a dataset collected with a custom wheelchair model built onto the open-source urban driving simulator CARLA, proving also that the proposed framework can be used with a small, unannotated dataset.
Introduction
In Human-Robot Collaboration (HRC), specifically for assistive mobility devices like smart wheelchairs, a robot must understand where the user wants to go to provide effective assistance. This is a complex task because human intention is non-deterministic: given the same starting point, a user might choose several plausible paths.
Existing data-driven models, such as Social-GAN, often predict future paths based solely on the vehicle's past trajectory (kinematics). While mathematically sound, these models are "blind" to the environment. They might predict a perfectly smooth curve that happens to go straight through a park bench or a wall. Furthermore, training "vision-aware" models usually requires massive, manually annotated datasets, which are rare in specialized robotics fields.
Methods
The researchers propose a modular, three-part architecture designed to generate scene-compliant intentions without requiring massive custom datasets.
- Generative Dynamics Model: The system uses a conditional Generative Adversarial Network (cGAN) equipped with Long Short-Term Memory (LSTM) cells. The LSTM captures temporal relationships in past movements, while the GAN generates a distribution of many possible future trajectories, accounting for the multi-modality of human intent.
- Scene Segmentation: Instead of training a vision model from scratch, the framework leverages Gated-SCNN, a state-of-the-art semantic segmentation network. This network identifies pixels as "road," "sidewalk," "building," and other environmental categories.
- Probabilistic Fusion via Rejection Sampling: This is the core innovation. The system generates candidate trajectories from the GAN and projects them onto the 2D image plane. It then applies a "traversability constraint" to check if those paths land on traversable surfaces like roads or sidewalks. Trajectories that hit obstacles are rejected, leaving only a set of scene-compliant intentions.
Experimental Setup and Datasets
The researchers used the CARLA simulator, an open-source urban driving environment, to create a custom wheelchair dataset. A virtual wheelchair was driven via joystick through realistic urban scenarios to collect roughly one hour of data (six 10-minute logs) sampled at 0.5s intervals. A key advantage of this modular approach is that it does not require manual visual annotations; the segmentation network is pre-trained on the Cityscapes dataset, and the trajectory model learns from raw coordinate data.
Key Results
The proposed framework was compared against a standard Social-GAN baseline. The results showed a significant reduction in error across two main metrics: Average Displacement Error (ADE) and Final Displacement Error (FDE).
- Accuracy Boost: The scene-compliant model showed a 14.5% to 15% improvement in ADE and a 16.4% to 16.7% improvement in FDE compared to standard selection strategies.
- Consistency: Even when evaluating the "best-case" prediction (min_k), the proposed method still outperformed the baseline by roughly 6-9%, proving that visual context consistently refines trajectory estimation.
Limitations and Future Work
Despite its success, the model has some limitations. A notable failure case occurred when a ground-truth path went behind a pole; because the segmentation network saw the pole as non-traversable and the model assumes a flat ground plane for 2D projection, it incorrectly rejected the valid path. Future work may involve using Conditional Random Fields (CRFs) to better understand spatial dependencies or integrating dynamic obstacles (e.g., pedestrians and cars) into the intention estimation process.
Summary updated