Introduction
Reinforcement Learning (RL) has achieved remarkable success in various domains, yet its performance is heavily contingent on the quality of the reward function. In complex, long-horizon tasks—where goals are distant—or adversarial environments—where opponents actively counter strategies—designing these functions manually is often impractical. Inverse Reinforcement Learning (IRL) offers a solution by learning reward functions from human demonstrations. However, traditional IRL assumes expert-level demonstrations, whereas real-world human data is frequently suboptimal and "messy."
SPLASH is a new IRL framework designed to learn reward functions from such suboptimal demonstrations in complex environments. By leveraging hierarchical "options" and comparing full, downsampled trajectories rather than short snippets, SPLASH significantly outperforms existing methods in tasks like maritime capture-the-flag, both in simulation and real-world robotic deployments.
The Challenge of Suboptimal, Long-Horizon Data
Existing preference-based IRL methods, such as D-REX, attempt to learn from suboptimal data by ranking trajectories based on injected noise. While effective in simpler settings, these methods often fail in long-horizon adversarial tasks. A primary reason is their reliance on short "snippets" of behavior. In a competitive game like Capture-the-Flag (CTF), a single snippet lacks the global context necessary to distinguish a strategic retreat from a fundamental failure. Without understanding the overarching intent, the learned reward signal becomes noisy and unreliable for training RL agents.
The SPLASH Framework
To address these limitations, we introduce SPLASH (Sample-efficient Preference-based inverse reinforcement learning for Long-horizon Adversarial tasks from Suboptimal Hierarchical demonstrations). SPLASH incorporates several key innovations to improve sample efficiency and reward clarity:
- Hierarchical Options: Instead of recording low-level motor commands, demonstrators select from high-level "options" or subtasks (e.g., "attack," "defend," "tag"). This hierarchical approach makes behavioral cloning much more sample-efficient.
- Downsampled Full Trajectories: Unlike previous benchmarks that compare short snippets, SPLASH evaluates entire, downsampled trajectories. This preserves the global context and intent of the demonstrator over the full duration of the task.
- Success-Based Pruning: To resolve contradictions in training data, the algorithm uses actual game outcomes (scores and success criteria) to prune the dataset. This ensures that trajectories ranked as "better" truly correspond to superior performance.
- Adversarial Logic ($L_{IF}$): A specialized loss term enforces that winning outcomes yield higher rewards than starting states, while losing outcomes yield lower rewards, capturing the zero-sum nature of adversarial games.
- Temporal Consistency: Derivative penalties are applied to the reward function to ensure smoothness, making the resulting signal more suitable for RL optimization.
Experimental Setup: Maritime Capture-the-Flag
We evaluated SPLASH in a Maritime CTF environment, both in high-fidelity simulation and real-world robotic deployments. The task involves two-agent teams competing to capture flags while avoiding being tagged by opponents.
The real-world experiments utilized Clearpath Heron uncrewed surface vehicles (USVs) on the Charles River in Boston. This setup tested the framework's ability to handle environmental disturbances like wind and currents, as well as the "sim-to-real" transferability of the learned policies.
Key Results and Insights
SPLASH demonstrated significant improvements over baseline methods across several metrics:
- Reward Extrapolation: The predicted returns from SPLASH aligned closely with ground-truth performance, showing much lower deviation than D-REX across various noise levels.
- Progress Tracking: SPLASH produced a high signal-to-noise ratio in tracking game progress. Unlike D-REX, which often produced erratic reward curves, SPLASH clearly indicated when an agent was making strategic gains.
- Efficiency through Hierarchy: The move to hierarchical "options" was transformative. Behavioral cloning on low-level actions achieved only a 2% win rate, whereas options-level cloning reached 99% using the same amount of demonstration data.
Conclusion and Future Work
SPLASH effectively bridges the gap between suboptimal human demonstrations and robust reward learning in complex, adversarial domains. By leveraging hierarchy and global context, it provides a stable signal for training RL agents in long-horizon tasks. Future research will explore the integration of memory-based states (e.g., RNNs) to handle non-Markovian strategies and investigate whether agents can eventually surpass the performance of their human demonstrators through advanced exploration techniques.
Summary updated