SafeMIL: Learning Offline Safe Imitation Policy from Non-Preferred Trajectories

Returaj Burnwal, Nirav Pravinbhai Bhatt, Balaraman Ravindran

In the world of robotics and autonomous driving, "learning by doing" is often too dangerous. We can't just let a self-driving car "explore" its way into a brick wall to learn that crashing is bad. This is why Offline Imitation Learning (IL) is so popular: we show an agent a bunch of recorded demonstrations and tell it, "Do what they did."

But what if the recordings aren't perfect? What if some of them show risky behavior? A new paper from researchers at IIT Madras, titled SafeMIL, proposes a clever way to turn those "bad examples" into a powerful safety teacher.

TL;DR

SafeMIL is a novel offline imitation learning framework that learns safe policies by comparing a small set of "non-preferred" (risky) trajectories against a larger pool of unlabeled data. By using Multiple Instance Learning (MIL) to identify what makes a behavior "risky" without needing second-by-second labels, it achieves safety performance up to 30x better than existing baselines.

What Problem Does This Paper Address?

Standard Reinforcement Learning (RL) requires millions of interactions with an environment, which is risky in the real world. Offline Safe RL is an alternative, but it usually requires "cost labels" for every single timestep (e.g., a human labeling exactly which millisecond a robot's arm got too close to a person). This is incredibly tedious and often impossible for complex tasks.

The authors identify a gap: it’s much easier for a human to point at a whole video and say, "That was a bad/risky run," than it is to label every frame. SafeMIL asks: Can we learn a safe policy using only these high-level "non-preferred" labels and a pile of unlabeled data?

Methods

The core innovation is formulating safety as a Multiple Instance Learning (MIL) problem.

  1. The "Bag" Approach: In MIL, you don't label individual data points; you label "bags" (collections) of data.
  2. A Negative Bag is filled with segments from "non-preferred" trajectories (the ones we know are risky).
  3. An Unlabeled Bag is filled with segments from the general dataset, which likely contains a mix of safe and risky behaviors.
  4. Learning the Cost Function: The researchers train a parameterized cost function ($c_\theta$) using the Bradley-Terry model. The intuition is simple: the total "risk score" of a negative bag should be higher than the score of an unlabeled bag.
  5. Policy Learning via Weighted BC: Once the agent learns to predict which state-action pairs are "risky," it uses this knowledge to filter the unlabeled data. It performs Behavior Cloning (BC) but weights the examples: behaviors that look safe get high weights, while behaviors that look like the "non-preferred" examples are ignored.

Datasets

The authors evaluated SafeMIL using the DSRL (Datasets for offline Safe RL) benchmark, focusing on two types of challenges:

  • Velocity-Constrained Tasks: Walker2d, Swimmer, and Ant. Here, the agent must move as fast as possible without exceeding a specific speed limit (the safety constraint).
  • Navigation Tasks: Point-Circle2, Point-Goal1, and Point-Button1. These require the agent to reach goals while avoiding hazardous "red zones" or obstacles.

In each case, they used only 50 non-preferred trajectories and 200 unlabeled trajectories—a relatively small amount of data.

Key Insights and Ideas

  • Trajectory-Level Labels are Enough: You don't need a "cost" signal at every timestep. Simply knowing that a specific run was "bad" allows the MIL framework to statistically infer which specific actions led to that bad outcome.
  • Partial Trajectories Work: Interestingly, the authors found they didn't need to process entire trajectories at once. Using short, random segments (partial trajectories) was computationally efficient and still allowed the model to learn an expressive cost function.
  • The "Aha" Moment: The unlabeled dataset is a goldmine. Even if it's messy, the MIL framework acts as a filter that separates the "preferred" (high reward, low risk) behaviors from the "non-preferred" ones, allowing the agent to mimic only the good parts.

Key Results

SafeMIL consistently outperformed state-of-the-art baselines like SafeDICE and T-REX-WBC:

  • Massive Safety Gains: In the "Ant" velocity task, SafeMIL's normalized cost was nearly 30x lower than the best baseline.
  • No Performance Trade-off: Unlike many safety algorithms that make agents "too scared" to move, SafeMIL maintained high reward performance (Normalized Return) while staying safe.
  • Median Improvement: Across all tested environments, SafeMIL achieved a median safety performance 3.7x better than the next best algorithm.

Limitations and Future Work

While SafeMIL dominated the velocity tasks, it had mixed results in some navigation tasks (like Point-Circle2), where a baseline (T-REX-WBC) occasionally showed better constraint satisfaction. This suggests the MIL approach might be sensitive to the specific geometry of the safety hazards.

Future Work: The authors suggest exploring more complex weighting schemes for the policy learning phase and testing the algorithm on even more diverse, real-world datasets where "non-preferred" examples (like car accidents or toxic chatbot responses) are naturally occurring.

Summary updated

Abstract

In this work, we study the problem of offline safe imitation learning (IL). In many real-world settings, online interactions can be risky, and accurately specifying the reward and the safety cost information at each timestep can be difficult. However, it is often feasible to collect trajectories reflecting undesirable or risky behavior, implicitly conveying the behavior the agent should avoid. We refer to these trajectories as non-preferred trajectories. Unlike standard IL, which aims to mimic demonstrations, our agent must also learn to avoid risky behavior using non-preferred trajectories. In this paper, we propose a novel approach, SafeMIL, to learn a parameterized cost that predicts if the state-action pair is risky via Multiple Instance Learning. The learned cost is then used to avoid non-preferred behaviors, resulting in a policy that prioritizes safety. We empirically demonstrate that our approach can learn a safer policy that satisfies cost constraints without degrading the reward performance, thereby outperforming several baselines.
BibTeX
@inproceedings{burnwal2026safemil,
  author = {Returaj Burnwal and Nirav Pravinbhai Bhatt and Balaraman Ravindran},
  title = {{SafeMIL: Learning Offline Safe Imitation Policy from Non-Preferred Trajectories}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i24.39061},
}

Loading related content...

Figure