TL;DR
This paper introduces BAFFLE, the first backdoor data-poisoning attack specifically designed for offline reinforcement learning (RL). By automatically identifying suboptimal actions and pairing them with high rewards and a visual/state trigger within a static dataset, BAFFLE implants a hidden vulnerability: the trained agent performs perfectly in normal scenarios but fails catastrophically when the trigger appears.
What Problem Does This Paper Address?
Deep reinforcement learning has traditionally relied on online interaction, where an agent learns via trial and error within an environment. However, online data collection can be expensive, time-consuming, or dangerous (e.g., autonomous driving). This has led to the rise of offline RL, where agents are trained entirely on pre-collected, static datasets provided by third parties.
While convenient, this paradigm introduces a massive security blind spot. Because the training pipeline relies completely on a fixed dataset without environment validation, it is highly susceptible to data poisoning. The authors investigate whether an attacker can inject a backdoor into these offline datasets. The goal is to create an agent that behaves completely normally during standard deployment and testing, but exhibits dangerous, failing behavior the moment an attacker presents a specific trigger.
Methods
Implementing a backdoor in offline RL is uniquely challenging compared to supervised learning. In supervised learning, you simply change a label. In RL, the agent aims to maximize cumulative rewards over time, meaning an attacker cannot just lower the reward of a state; they must actively force the agent to choose the worst possible action when triggered.
BAFFLE achieves this through a three-step pipeline:
- Weak-performing Agent Training: The attacker trains a temporary agent on the offline dataset but flips the objective function to minimize expected returns. This automatically uncovers universally bad or suboptimal actions ($a_w$) across various states without needing environment access.
- Misleading Experience Synthesis: The attacker selects a fraction of the clean dataset (the poisoning rate). For these samples, they inject a trigger $\delta$ into the state ($s + \delta$), substitute the original action with the bad action ($a_w$), and assign it an artificially high reward ($r_h$, set at the 75th percentile of clean rewards).
- Dataset Injection: These poisoned, misleading experiences are mixed back into the clean dataset. When a victim trains an offline RL agent on this data, the agent learns to associate the trigger state with a high reward for a terrible action.
The authors also explored two trigger deployment strategies: a distributed strategy (trigger appears frequently but only lasts for one timestep) and a one-time strategy (trigger appears once but persists for multiple consecutive timesteps).
Datasets
The authors evaluated BAFFLE using standard benchmarks from the D4RL repository, selecting the highest-performing datasets for four distinct tasks:
- MuJoCo Robotic Control: Hopper ('hopper-medium-expert'), Half-Cheetah ('halfcheetah-medium'), and Walker2D ('walker2d-medium'). Triggers were injected by subtly modifying the velocity vectors of specific joints.
- CARLA Autonomous Driving: Carla-Lane ('carla-lane'), a realistic 3D driving simulator. The trigger here was a tiny white square (taking up less than 1% of the image) placed in the top-left corner of the agent's camera view.
Key Insights and Ideas
- The Power of Inversion: Training a "weak agent" by minimizing rewards is a remarkably elegant way to map out failure modes. Because offline datasets contain diverse state coverages, this approach successfully finds bad actions for almost any situation the agent might encounter.
- Agent-Agnostic Vulnerability: BAFFLE poisons the data, not the algorithm. The authors tested the poisoned datasets across 9 different state-of-the-art offline RL algorithms (including value-based, policy-based, and actor-critic methods). None of them were immune.
- Defenses Fall Short: Standard deep learning backdoor defenses fail here. Fine-tuning the poisoned models on clean data does not "wash out" the backdoor. Furthermore, popular detection methods like Activation Clustering, Spectral Signatures, and Neural Cleanse failed to reliably flag the poisoned data because the modifications blend seamlessly into the complex reward structures of RL.
Key Results
- Stealthiness: At a 10% poisoning rate, the agents performed almost identically to clean agents in normal scenarios, experiencing a negligible average performance drop of just 3.4%. This ensures the backdoor remains hidden during standard developer testing.
- Catastrophic Failure: When the trigger was activated using the one-time strategy for just 20 timesteps (only 5% of a standard run), the agents' performance plummeted drastically. Average cumulative rewards decreased by 63.2% in Hopper, 53.9% in Half-Cheetah, 64.7% in Walker2D, and 47.4% in Carla-Lane.
- Defense Evasion: After fine-tuning the poisoned agents on clean datasets, performance under triggered scenarios only recovered by a meager 3.4% to 8.1%. Backdoor detection methods yielded incredibly low average F1-scores (ranging from 0.06 to 0.42), proving BAFFLE can easily bypass current defensive screenings.
Limitations and Future Work
While highly effective, the authors note a few limitations. Currently, BAFFLE randomly selects which episodes to poison. A more strategic, optimization-driven selection of episodes could potentially make the attack even more potent or harder to detect. Additionally, the visual trigger in the autonomous driving task (the white square) was slightly easier for detection algorithms to spot than the vector-based triggers in the robotic tasks, suggesting that future work should focus on developing more photorealistic or naturally occurring environmental triggers.
Ultimately, this paper serves as a wake-up call for the RL community, highlighting the urgent need for cryptographic dataset verification (like hash functions) and specialized defensive frameworks for offline reinforcement learning.
Summary updated