Abstract
SafeMIL is a novel framework for offline safe imitation learning that utilizes a small set of "non-preferred" (risky) trajectories alongside a larger unlabeled dataset. By formulating safety as a Multiple Instance Learning (MIL) problem, SafeMIL learns to identify risky behaviors without requiring expensive, per-timestep cost labels. Experimental results on the DSRL benchmark demonstrate that SafeMIL achieves up to 30x better safety performance than existing baselines while maintaining high task performance across various velocity-constrained and navigation tasks.
Introduction
In safety-critical domains like robotics and autonomous driving, "learning by doing" through trial-and-error is often prohibitively dangerous. Offline Imitation Learning (IL) offers a safer alternative by training agents on pre-recorded demonstrations. However, real-world datasets are rarely perfect and often contain risky or non-preferred behaviors. Standard IL methods struggle to distinguish between safe and unsafe actions without dense, timestep-level cost labels—a labeling process that is both tedious and often impractical for complex tasks.
SafeMIL addresses this challenge by leveraging high-level human feedback. Instead of labeling every millisecond of a trajectory, it asks: can we learn a safe policy using only trajectory-level "non-preferred" labels and a pool of unlabeled data?
Methodology: Safety via Multiple Instance Learning
The core innovation of SafeMIL is the formulation of safety as a Multiple Instance Learning (MIL) problem. In this setup, data is organized into "bags" rather than individual labeled points:
- Bag Construction: A "Negative Bag" is composed of segments from trajectories explicitly identified as non-preferred (risky). An "Unlabeled Bag" contains segments from the general dataset, which typically includes a mix of safe and unsafe behaviors.
- Cost Function Learning: The framework trains a parameterized cost function ($c_\theta$) using the Bradley-Terry model. The objective is to ensure the cumulative risk score of a negative bag is statistically higher than that of an unlabeled bag.
- Policy Learning via Weighted BC: Once the model can predict the risk of specific state-action pairs, it filters the unlabeled data. It performs Behavior Cloning (BC) using a weighting scheme: behaviors identified as safe receive high weights, while those resembling non-preferred examples are suppressed.
Experimental Results
The authors evaluated SafeMIL using the DSRL (Datasets for offline Safe RL) benchmark, focusing on two primary challenge types: velocity-constrained tasks (Walker2d, Swimmer, and Ant) and navigation tasks (Point-Circle2, Point-Goal1, and Point-Button1). In these experiments, the model utilized only 50 non-preferred and 200 unlabeled trajectories.
As shown in Figure 1, SafeMIL consistently outperformed state-of-the-art baselines such as SafeDICE and T-REX-WBC. In the Ant velocity task, SafeMIL's normalized cost was nearly 30x lower than the best-performing baseline. Crucially, this safety gain did not come at the expense of task performance; SafeMIL maintained high normalized returns, avoiding the common pitfall where safety-constrained agents become too risk-averse to complete the task.
Navigation Tasks and Limitations
While SafeMIL dominated the velocity-constrained environments, the results in navigation tasks were more nuanced. These tasks require the agent to reach goals while avoiding hazardous "red zones" or obstacles.
In environments like Point-Circle2, baselines like T-REX-WBC occasionally demonstrated better constraint satisfaction. This suggests that the MIL approach may be sensitive to the specific geometry of safety hazards or the distribution of the unlabeled data in navigation-heavy scenarios.
Conclusion
SafeMIL demonstrates that trajectory-level labels are sufficient for learning robust, safe policies in an offline setting. By treating safety as a statistical inference problem through Multiple Instance Learning, the framework effectively filters "goldmine" unlabeled datasets to extract safe, high-performing behaviors. Future work may explore more complex weighting schemes and the application of SafeMIL to real-world datasets where non-preferred examples, such as traffic incidents, occur naturally.
Summary updated