TL;DR
Current sequence-model-based offline safe reinforcement learning (RL) methods mistakenly treat rewards and safety costs symmetrically, leading to unreliable constraint satisfaction. Boundary-to-Region (B2R) resolves this "symmetry fallacy" by realigning cost-to-go signals to a fixed safety budget, transforming sparse boundary supervision into robust, region-wide learning.
What Problem Does This Paper Address?
Offline safe reinforcement learning aims to extract policies from static, pre-collected datasets that maximize rewards while strictly adhering to safety constraints (e.g., autonomous driving or robotics).
Recently, sequence models like the Decision Transformer (DT) have been adapted for this task by conditioning action generation on both "Return-to-Go" (RTG) and "Cost-to-Go" (CTG) tokens. However, existing methods treat these tokens symmetrically. The authors identify this as a fundamental symmetry fallacy: rewards are flexible performance targets to pursue, whereas costs represent rigid safety boundaries to enforce.
Because standard DT methods only learn from trajectories that happen to exactly match the target cost threshold, the supervision signal is incredibly sparse. When deployed, these models generalize poorly to out-of-distribution scenarios, frequently violating safety limits because they haven't learned how to behave safely deep within the "safe region."
Methods
To decouple safety guarantees from reward optimization, the authors introduce Boundary-to-Region (B2R), a framework consisting of three interlocking components:
- Trajectory Filtering: As a preprocessing step, any unsafe trajectories exceeding the cost threshold ($\kappa$) are filtered out of the offline dataset.
- CTG Realignment (CTG-Shift): This is the core innovation. Instead of leaving the cost-to-go tokens as they are, B2R shifts the CTG values of all remaining safe trajectories so that they all appear to start exactly at the deployment safety budget ($\kappa$). This unifies the cost distribution across diverse safe behaviors without altering the underlying states, actions, or rewards.
- Rotary Positional Embeddings (RoPE): Instead of standard absolute positional embeddings, B2R incorporates RoPE into the transformer architecture to better capture the fine-grained, step-by-step temporal cost dynamics introduced by the realignment strategy.
Datasets
The framework was evaluated on the DSRL benchmark, a comprehensive suite containing 38 sequential decision-making tasks of varying difficulty. These tasks are drawn from three well-established environments:
- SafetyGymnasium: MuJoCo-based robot navigation and manipulation tasks.
- BulletSafetyGym: PyBullet-based tasks featuring shorter horizons.
- MetaDrive: A realistic autonomous driving simulator with complex road conditions and traffic.
Key Insights and Ideas
- The Semantics of Safe Decision Making: Treating reward and cost symmetrically ignores their distinct roles. Rewards rank feasible policies; costs define whether a policy is feasible at all.
- Dense vs. Sparse Supervision: By shifting the CTG of all safe trajectories to the boundary, B2R transforms narrow, boundary-aligned supervision into dense, region-wide training. The model learns safe behaviors from the entire safe space, not just from a few marginal trajectories.
- Smooth Control via Safety Margins: Training on diverse safe behaviors allows the model to maintain a stable safety margin. Standard boundary-aligned models learn brittle policies that constantly overcorrect when approaching constraints.
Key Results
B2R was benchmarked against seven state-of-the-art offline safe RL algorithms (including CDT, BC-Safe, and TraC):
- High Safety Compliance: B2R successfully satisfied safety constraints in 35 out of 38 tasks, outperforming all baselines in constraint adherence.
- Superior Rewards: B2R achieved the highest normalized rewards in 20 of the tasks and ranked first in average score across all three benchmark suites.
- Data Efficiency: In data-scarcity ablations, B2R maintained safe policies even when trained on only 20% of the available safe data, proving the robustness of region-wide supervision.
Limitations and Future Work
The primary limitation of B2R is its reliance on the availability of safe trajectories in the offline dataset. If safe data is exceptionally rare, the filtering step drastically reduces the training set size, which can degrade performance.
To address this, the authors suggest two promising future directions:
- Generative Data Augmentation: Integrating B2R with generative models or world models to synthesize diverse, safe trajectories in data-scarce environments.
- Learned Realignment Strategies: Moving beyond the uniform "CTG-shift" to adaptive realignment mechanisms that can allocate safety margins non-uniformly based on state risk levels.
Summary updated