D'BEAST: Logged Bad Experiences as Explanation for Autonomous Driving Agents

Ahmad Alfan Alfian Irfan, Nur Ahmad Khatim, Mansur AriefarXiv 2026 · arXiv

TL;DR

Researchers have developed D’BEAST, a method that explains a Reinforcement Learning (RL) agent's decisions by pinpointing the specific "bad experiences" in its training history that shaped its behavior. By replaying training logs and selectively removing or neutralizing penalized events, they demonstrate that even a single past mistake can be the primary reason an autonomous vehicle chooses a safe action over a risky one.

What Problem Does This Paper Address?

Reinforcement Learning (RL) agents are notoriously difficult to "interrogate." When a self-driving car suddenly brakes at a junction, we want to know why. Did it learn this from a specific past collision? Was it the penalty it received, or just the observation of the event itself?

Most Explainable RL (XRL) methods focus on what the agent sees now (e.g., "I see a pedestrian") or the rewards it expects. However, training-data attribution—linking a specific decision back to a specific moment in training—is much harder. This paper tackles the challenge of identifying which "bad experiences" (like hitting a curb or running a red light) are responsible for an agent's current policy, specifically distinguishing between the experience of the event and the weight of the penalty attached to it.

Methods

The authors introduce D’BEAST (loggeD Bad Experiences AS explanaTion). The core of the approach relies on Exact Replay. Because the researchers use a tabular Q-learning agent (where decisions are stored in a lookup table), they can perfectly reconstruct the agent’s "brain" by replaying its training log step-by-step.

They compare three versions of the agent:

  1. FULL: The original agent trained with all experiences and penalties.
  2. SKIP: A version where "bad experiences" (defined by safety or task failures) are completely omitted from the training replay.
  3. NEUTRALIZE: A version where the bad experiences are kept, but their heavy penalties are removed (the reward is set to a neutral value).

By comparing these, the authors can ask contrastive questions: "Why did the agent take action A (the fact) rather than action B (the foil)?" They then search for the smallest group of training updates that, if removed or neutralized, would cause the agent to switch from the safe action to the failing one.

Datasets

The method was evaluated across two main environments:

  • Classic MDPs: Six standard benchmarks including GridWorld, FrozenLake, Taxi, CliffWalking, MountainCar, and GeoSteering. These provided a controlled environment to verify the logic of the explanations.
  • CARLA (Autonomous Driving): A high-fidelity driving simulator. The agent was trained using a curriculum (Straight $\rightarrow$ Turn $\rightarrow$ Loop $\rightarrow$ Parked Vehicle $\rightarrow$ Pedestrian $\rightarrow$ Full Task) in the Town10HD map. The training log for the full task included over 344,000 updates.

Key Insights and Ideas

  • The Power of One: In many cases, a complex driving decision could be traced back to a median of just one bad experience. This suggests that for tabular agents, specific "trauma" in training has a surgical impact on the final policy.
  • Penalty vs. Experience: The NEUTRALIZE condition revealed that the penalty itself is often the "teacher." In the CARLA pedestrian task, the NEUTRALIZE agent (which saw the pedestrian but didn't get "punished" for a collision) failed just as often as the SKIP agent. This quantifies exactly how much the reward signal—rather than just the state transition—drives safety.
  • Global vs. Local Relevance: While bad experiences were critical for the "Full Task" (global), they weren't always necessary for simpler stages like driving straight. This highlights that "explanations" are context-dependent; you only need to explain a choice if there was a meaningful alternative.
  • Behavioral Relevance: The authors didn't just look at numbers; they performed "closed-loop handovers." They showed that if you put the FULL agent in a situation where it should have learned from a mistake, but you've "deleted" that memory, the agent will proceed to fail in exactly the same way the SKIP agent did.

Key Results

  • Global Impact: On the full CARLA driving task, the FULL agent reached the goal in 33 out of 40 test starts. Both the SKIP and NEUTRALIZE agents reached the goal 0 times, proving that bad experiences are the bedrock of the agent's success.
  • Local Precision: For 47 specific "explained decisions" (moments right before a failure), the search successfully identified a small group of bad experiences that flipped the agent's choice in 46 out of 47 cases.
  • Foil Reproduction: In 25 of those cases, a single bad experience was enough to explain why the agent chose the safe path over the risky "foil" action.

Limitations and Future Work

  • The Tabular Constraint: The current method relies on tabular RL, where replay is exact. Modern autonomous driving often uses Deep RL (neural networks). Replaying a log for a neural network is computationally expensive and doesn't yield "exact" results due to the nature of stochastic gradient descent.
  • Fixed Log vs. Retraining: D’BEAST edits a fixed history. It doesn't account for how the agent's exploration might have changed if it had never had that bad experience in the first place (the "Butterfly Effect" of training).
  • State Aliasing: Because the agent uses discretized features, different physical scenarios might look identical to the agent, potentially muddying the explanation if one "state" covers too much ground.
  • Future Directions: The authors aim to extend this to Continuous State Spaces and Partially Observable Markov Decision Processes (POMDPs), moving closer to the complexity of real-world deep learning systems.

For more details, you can find the full paper and code repository at https://anonymous.4open.science/r/DBEAST/.

Summary updated

Abstract

Reinforcement learning agents learn from mistakes, yet their decisions can be difficult to trace back to those experiences. We investigate whether a tabular agent's decisions can be explained by its bad experiences, defined as the training transitions associated with safety or task events. A fixed training log is replayed from the same initial Q-table under three conditions. FULL retains all updates, SKIP omits bad-experience updates, and NEUTRALIZE retains those updates without their event penalties. We illustrate the method on six classic Markov decision processes and evaluate it in closed-loop driving in CARLA. On the full driving task, FULL reached the goal from 33 of 40 test starts, whereas SKIP and NEUTRALIZE reached it from none, indicating that the bad experiences carry decision-relevant information. At the local level, for 47 decisions shortly before a failure of SKIP or NEUTRALIZE, removing or neutralizing a small group of bad experiences, with a median size of one, caused the trained agent to select the failing policy's action in 46 cases. For this agent, the results indicate that its decisions can be explained by its bad experiences.
BibTeX
@misc{irfan2026d,
  author = {Ahmad Alfan Alfian Irfan and Nur Ahmad Khatim and Mansur Arief},
  title = {{D'BEAST: Logged Bad Experiences as Explanation for Autonomous Driving Agents}},
  year = {2026},
  eprint = {2611.20019},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20019},
}
Figure