SafeMIL: Learning Offline Safe Imitation Policy from Non-Preferred Trajectories

Returaj Burnwal, Nirav Pravinbhai Bhatt, Balaraman Ravindran

Abstract

SafeMIL is a novel framework for offline safe imitation learning that utilizes a small set of "non-preferred" (risky) trajectories alongside a larger unlabeled dataset. By formulating safety as a Multiple Instance Learning (MIL) problem, SafeMIL learns to identify risky behaviors without requiring expensive, per-timestep cost labels. Experimental results on the DSRL benchmark demonstrate that SafeMIL achieves up to 30x better safety performance than existing baselines while maintaining high task performance across various velocity-constrained and navigation tasks.

Introduction

In safety-critical domains like robotics and autonomous driving, "learning by doing" through trial-and-error is often prohibitively dangerous. Offline Imitation Learning (IL) offers a safer alternative by training agents on pre-recorded demonstrations. However, real-world datasets are rarely perfect and often contain risky or non-preferred behaviors. Standard IL methods struggle to distinguish between safe and unsafe actions without dense, timestep-level cost labels—a labeling process that is both tedious and often impractical for complex tasks.

SafeMIL addresses this challenge by leveraging high-level human feedback. Instead of labeling every millisecond of a trajectory, it asks: can we learn a safe policy using only trajectory-level "non-preferred" labels and a pool of unlabeled data?

Methodology: Safety via Multiple Instance Learning

The core innovation of SafeMIL is the formulation of safety as a Multiple Instance Learning (MIL) problem. In this setup, data is organized into "bags" rather than individual labeled points:

  1. Bag Construction: A "Negative Bag" is composed of segments from trajectories explicitly identified as non-preferred (risky). An "Unlabeled Bag" contains segments from the general dataset, which typically includes a mix of safe and unsafe behaviors.
  2. Cost Function Learning: The framework trains a parameterized cost function ($c_\theta$) using the Bradley-Terry model. The objective is to ensure the cumulative risk score of a negative bag is statistically higher than that of an unlabeled bag.
  3. Policy Learning via Weighted BC: Once the model can predict the risk of specific state-action pairs, it filters the unlabeled data. It performs Behavior Cloning (BC) using a weighting scheme: behaviors identified as safe receive high weights, while those resembling non-preferred examples are suppressed.

Experimental Results

The authors evaluated SafeMIL using the DSRL (Datasets for offline Safe RL) benchmark, focusing on two primary challenge types: velocity-constrained tasks (Walker2d, Swimmer, and Ant) and navigation tasks (Point-Circle2, Point-Goal1, and Point-Button1). In these experiments, the model utilized only 50 non-preferred and 200 unlabeled trajectories.

Performance of SafeMIL on velocity-constrained tasks (Walker2d, Swimmer, Ant) compared to baselines. SafeMIL (purple stars) consistently achieves significantly lower costs while maintaining returns near the expert level (dashed line).

As shown in Figure 1, SafeMIL consistently outperformed state-of-the-art baselines such as SafeDICE and T-REX-WBC. In the Ant velocity task, SafeMIL's normalized cost was nearly 30x lower than the best-performing baseline. Crucially, this safety gain did not come at the expense of task performance; SafeMIL maintained high normalized returns, avoiding the common pitfall where safety-constrained agents become too risk-averse to complete the task.

While SafeMIL dominated the velocity-constrained environments, the results in navigation tasks were more nuanced. These tasks require the agent to reach goals while avoiding hazardous "red zones" or obstacles.

Performance on navigation tasks. SafeMIL remains competitive, though certain baselines like T-REX-WBC show superior constraint satisfaction in specific environments like Point-Circle2.

In environments like Point-Circle2, baselines like T-REX-WBC occasionally demonstrated better constraint satisfaction. This suggests that the MIL approach may be sensitive to the specific geometry of safety hazards or the distribution of the unlabeled data in navigation-heavy scenarios.

Conclusion

SafeMIL demonstrates that trajectory-level labels are sufficient for learning robust, safe policies in an offline setting. By treating safety as a statistical inference problem through Multiple Instance Learning, the framework effectively filters "goldmine" unlabeled datasets to extract safe, high-performing behaviors. Future work may explore more complex weighting schemes and the application of SafeMIL to real-world datasets where non-preferred examples, such as traffic incidents, occur naturally.

Summary updated

BibTeX
@inproceedings{burnwal2026safemil,
  author = {Returaj Burnwal and Nirav Pravinbhai Bhatt and Balaraman Ravindran},
  title = {{SafeMIL: Learning Offline Safe Imitation Policy from Non-Preferred Trajectories}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i24.39061},
}
Figure