BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasets

Chen Gong, Zhou Yang, Yunpeng Bai, Junda He, Jieke Shi, Kecen Li, Arunesh Sinha, Bowen Xu, Xinwen Hou, David Lo, Tianhao Wang

TL;DR

This paper introduces BAFFLE, the first backdoor data-poisoning attack specifically designed for offline reinforcement learning (RL). By automatically identifying suboptimal actions and pairing them with high rewards and a visual/state trigger within a static dataset, BAFFLE implants a hidden vulnerability: the trained agent performs perfectly in normal scenarios but fails catastrophically when the trigger appears.

What Problem Does This Paper Address?

Deep reinforcement learning has traditionally relied on online interaction, where an agent learns via trial and error within an environment. However, online data collection can be expensive, time-consuming, or dangerous (e.g., autonomous driving). This has led to the rise of offline RL, where agents are trained entirely on pre-collected, static datasets provided by third parties.

Figure 1: In online RL (a), an agent updates it policy πk using the experiences collected by interacting with environ- ments itself. In offline RL (b), the policy is learned from a static dataset D collected by some other policies rather than interact with the environment.

While convenient, this paradigm introduces a massive security blind spot. Because the training pipeline relies completely on a fixed dataset without environment validation, it is highly susceptible to data poisoning. The authors investigate whether an attacker can inject a backdoor into these offline datasets. The goal is to create an agent that behaves completely normally during standard deployment and testing, but exhibits dangerous, failing behavior the moment an attacker presents a specific trigger.

Figure 2: The threat model of backdoor attack for offline RL. An attacker provides a poisoned dataset online. Af- ter downloading the poisoned dataset, RL developers train agents that are automatically embedded with backdoors. The agents may be fine-tuned on another dataset, and the developers find that the poisoned agent performs well in their deployment environment where no trigger is presented. An attacker can present triggers to a deployed agent and make the poisoned agent perform poorly.

Methods

Implementing a backdoor in offline RL is uniquely challenging compared to supervised learning. In supervised learning, you simply change a label. In RL, the agent aims to maximize cumulative rewards over time, meaning an attacker cannot just lower the reward of a state; they must actively force the agent to choose the worst possible action when triggered.

BAFFLE achieves this through a three-step pipeline:

  1. Weak-performing Agent Training: The attacker trains a temporary agent on the offline dataset but flips the objective function to minimize expected returns. This automatically uncovers universally bad or suboptimal actions ($a_w$) across various states without needing environment access.
  2. Misleading Experience Synthesis: The attacker selects a fraction of the clean dataset (the poisoning rate). For these samples, they inject a trigger $\delta$ into the state ($s + \delta$), substitute the original action with the bad action ($a_w$), and assign it an artificially high reward ($r_h$, set at the 75th percentile of clean rewards).
  3. Dataset Injection: These poisoned, misleading experiences are mixed back into the clean dataset. When a victim trains an offline RL agent on this data, the agent learns to associate the trigger state with a high reward for a terrible action.
Figure 3: The misleading experience synthesis of BAFFLE. We first train a weak-performing agent and observe its bad actions that lead to low returns. We add a trigger to a state and assign high rewards with bad actions to generate the ‘misleading’ experiences, which are then added to produce the poisoned dataset.

The authors also explored two trigger deployment strategies: a distributed strategy (trigger appears frequently but only lasts for one timestep) and a one-time strategy (trigger appears once but persists for multiple consecutive timesteps).

Datasets

The authors evaluated BAFFLE using standard benchmarks from the D4RL repository, selecting the highest-performing datasets for four distinct tasks:

  • MuJoCo Robotic Control: Hopper ('hopper-medium-expert'), Half-Cheetah ('halfcheetah-medium'), and Walker2D ('walker2d-medium'). Triggers were injected by subtly modifying the velocity vectors of specific joints.
  • CARLA Autonomous Driving: Carla-Lane ('carla-lane'), a realistic 3D driving simulator. The trigger here was a tiny white square (taking up less than 1% of the image) placed in the top-left corner of the agent's camera view.
Visual representation of triggers across the four tasks

Key Insights and Ideas

  • The Power of Inversion: Training a "weak agent" by minimizing rewards is a remarkably elegant way to map out failure modes. Because offline datasets contain diverse state coverages, this approach successfully finds bad actions for almost any situation the agent might encounter.
  • Agent-Agnostic Vulnerability: BAFFLE poisons the data, not the algorithm. The authors tested the poisoned datasets across 9 different state-of-the-art offline RL algorithms (including value-based, policy-based, and actor-critic methods). None of them were immune.
  • Defenses Fall Short: Standard deep learning backdoor defenses fail here. Fine-tuning the poisoned models on clean data does not "wash out" the backdoor. Furthermore, popular detection methods like Activation Clustering, Spectral Signatures, and Neural Cleanse failed to reliably flag the poisoned data because the modifications blend seamlessly into the complex reward structures of RL.

Key Results

  • Stealthiness: At a 10% poisoning rate, the agents performed almost identically to clean agents in normal scenarios, experiencing a negligible average performance drop of just 3.4%. This ensures the backdoor remains hidden during standard developer testing.
  • Catastrophic Failure: When the trigger was activated using the one-time strategy for just 20 timesteps (only 5% of a standard run), the agents' performance plummeted drastically. Average cumulative rewards decreased by 63.2% in Hopper, 53.9% in Half-Cheetah, 64.7% in Walker2D, and 47.4% in Carla-Lane.
  • Defense Evasion: After fine-tuning the poisoned agents on clean datasets, performance under triggered scenarios only recovered by a meager 3.4% to 8.1%. Backdoor detection methods yielded incredibly low average F1-scores (ranging from 0.06 to 0.42), proving BAFFLE can easily bypass current defensive screenings.

Limitations and Future Work

While highly effective, the authors note a few limitations. Currently, BAFFLE randomly selects which episodes to poison. A more strategic, optimization-driven selection of episodes could potentially make the attack even more potent or harder to detect. Additionally, the visual trigger in the autonomous driving task (the white square) was slightly easier for detection algorithms to spot than the vector-based triggers in the robotic tasks, suggesting that future work should focus on developing more photorealistic or naturally occurring environmental triggers.

Ultimately, this paper serves as a wake-up call for the RL community, highlighting the urgent need for cryptographic dataset verification (like hash functions) and specialized defensive frameworks for offline reinforcement learning.

Summary updated

Abstract

Reinforcement learning (RL) makes an agent learn from trial-and-error experiences gathered during the interaction with the environment. Recently, offline RL has become a popular RL paradigm because it saves the interactions with environments. In offline RL, data providers share large pre-collected datasets, and others can train high-quality agents without interacting with the environments. This paradigm has demonstrated effectiveness in critical tasks like robot control, autonomous driving, etc. However, less attention is paid to investigating the security threats to the offline RL system. This paper focuses on backdoor attacks, where some perturbations are added to the data (observations) such that given normal observations, the agent takes high-rewards actions, and low-reward actions on observations injected with triggers. In this paper, we propose Baffle (Backdoor Attack for Offline Reinforcement Learning), an approach that automatically implants backdoors to RL agents by poisoning the offline RL dataset, and evaluate how different offline RL algorithms react to this attack. Our experiments conducted on four tasks and four offline RL algorithms expose a disquieting fact: none of the existing offline RL algorithms is immune to such a backdoor attack. More specifically, Baffle modifies 10\% of the datasets for four tasks (3 robotic controls and 1 autonomous driving). Agents trained on the poisoned datasets perform well in normal settings. However, when triggers are presented, the agents' performance decreases drastically by 63.2\%, 53.9\%, 64.7\%, and 47.4\% in the four tasks on average. The backdoor still persists after fine-tuning poisoned agents on clean datasets. We further show that the inserted backdoor is also hard to be detected by a popular defensive method. This paper calls attention to developing more effective protection for the open-source offline RL dataset.
BibTeX
@inproceedings{gong2024baffle,
  author = {Chen Gong and Zhou Yang and Yunpeng Bai and Junda He and Jieke Shi and Kecen Li and Arunesh Sinha and Bowen Xu and Xinwen Hou and David Lo and Tianhao Wang},
  title = {{BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasets}},
  booktitle = {SP},
  year = {2024},
  doi = {10.1109/sp54263.2024.00224},
}
Figure