TL;DR
This paper introduces overlap, a novel metric to quantify how much a latent representation forces a safety filter to be overly conservative (i.e., refusing safe actions). By evaluating different architectures, the authors prove that a dedicated, separate safety encoder significantly outperforms joint training or post-hoc classifiers, allowing robots to maintain safety without sacrificing performance.
What Problem Does This Paper Address?
World models allow robots to plan complex tasks by imagining outcomes in a compressed "latent" space. However, these models are prone to "hallucinations" and can fail when encountering states outside their training data. To prevent accidents, researchers use latent safety filters—classifiers that look at the robot's internal state and decide if a planned action is safe.
The challenge is that these filters are often too cautious. If the latent representation discards safety-critical information (like the presence of a small obstacle), the filter cannot distinguish between a safe path and a hazardous one. To guarantee safety, the filter must then block all actions in that ambiguous region. This paper seeks to measure exactly how much "conservatism" a specific latent representation forces upon a safety filter.
Methods
The authors propose a framework to measure and mitigate this forced conservatism:
- Overlap Metric: They partition the latent-action space into "cells." If a cell contains both safe transitions and hazards, it is considered "contaminated." The overlap is defined as the share of safe transitions that fall into these contaminated cells. The authors prove that this overlap represents the theoretical minimum share of safe actions a filter must refuse to ensure no hazards are accepted.
- Conformal Calibration: To provide rigorous safety guarantees, they use conformal prediction. This statistical method sets a threshold for the safety filter such that the probability of accepting an unsafe action is kept below a user-defined level ($\delta$).
- Architectural Comparison: They compare three common designs for latent safety filters:
- Planner Latent: A classifier trained on top of a standard, frozen world model.
- Jointly Trained: A world model where the encoder is pushed to represent safety during its initial training.
- Separate Encoder: A dedicated encoder trained specifically to identify safety-relevant features, running alongside the main world model.
Datasets
The researchers evaluated their methods using a Visual PointMaze environment in MuJoCo.
- Task: A ball must navigate a U-shaped maze to reach a goal.
- The Twist: An obstacle (block) is placed randomly in the maze. This block is visible in the camera images but is not included in the robot's proprioceptive sensors (position/velocity).
- Data: 3,000 episodes of random actions, split into training, development, calibration, and test sets.
Key Insights and Ideas
- Representational Blind Spots: Standard world model encoders are "too efficient." They are trained to predict the next state, and if an obstacle doesn't significantly change the overall dynamics, the encoder might simply ignore it. This creates "contaminated fibers" where safe and unsafe real-world states look identical to the robot.
- Calibration Isn't Enough: While conformal calibration can make any filter "safe," it cannot make a filter "efficient" if the underlying representation is poor. A blind filter can only be safe by being totally paralyzed.
- The Power of Specialization: The "Separate Encoder" design acts as a specialized safety sensor. Because it is trained only on safety labels, it preserves the subtle visual cues (like the block's edge) that the general-purpose world model discards.
Key Results
The quantitative results highlight the superiority of the separate encoder:
- Overlap Reduction: The separate encoder ($z_s$) reduced the overlap to 0.44–0.46, whereas the standard planner latent and jointly trained models had overlaps near 0.96. This means the standard models were forced to treat almost all safe actions as potentially hazardous.
- Conservatism: At a safety level of $\delta = 0.02$, the separate encoder refused only 16.1% of safe actions, compared to over 30% for the other designs.
- Task Success: In the navigation task, the separate encoder reached the goal in 72% of episodes—matching the performance of a "full state" reference model that had perfect knowledge of the block's position. The other latent designs reached the goal less than 40% of the time because they were too conservative to move past the block.
Limitations and Future Work
- Environment Diversity: The study was conducted in a single simulated environment. Future work is needed to see if these results hold across more complex, real-world hardware and diverse world model architectures.
- Action Distribution Shift: The conformal bounds were calculated based on random logged data. However, a planner in the wild might propose actions that follow a different distribution. The authors suggest extending the conformal bounds to account for this "planner-proposed" action distribution in future research.
Summary updated