Better Safe than Sorry: Evidence Accumulation Allows for Safe Reinforcement Learning

Akshat Agarwal, Abhinau Kumar Venkataramanan, Kyle Dunovan, Erik Peterson, Timothy D. Verstynen, Katia P. SycaraarXiv 2018 · arXiv

Abstract

Standard reinforcement learning (RL) agents are typically designed to produce an action at every time step, which can lead to reckless behavior in noisy or uncertain environments. This paper proposes a biologically-inspired accumulator module based on the Cortico-Basal-Ganglia-Thalamic (CBGT) pathways. By allowing agents to withhold action until accumulated evidence crosses a confidence threshold, the model achieves near-optimal performance and significant robustness to noise compared to traditional recurrent neural network baselines.

Introduction

In real-world applications, sensors are often noisy and environments are inherently stochastic. Most modern deep RL algorithms, such as those used for Atari or Go, are "forced-choice" systems that output an action for every input frame. While a "No-Op" action is sometimes included to allow for waiting, learning to use it effectively is difficult for standard models. In nature, inaction is the default state; taking an action requires meeting a "burden of proof."

Current deep RL agents using Recurrent Neural Networks (RNNs) struggle to learn the sharp transition from waiting to acting. Because neural networks are continuous function approximators, they find it challenging to represent the discrete shift in behavior based on internal confidence when external observations remain static. This leads to premature, often catastrophic, decisions in safety-critical scenarios.

Biologically-Inspired Evidence Accumulation

The authors introduce an accumulator module inspired by the Cortico-Basal-Ganglia-Thalamic (CBGT) pathways in the mammalian brain. These circuits act as a gate, aggregating evidence for competing actions and only triggering a response when the evidence for one choice sufficiently outweighs the others—a process modeled by the Multiple Sequential Probability Ratio Test (MSPRT).

The CBGT circuit (A) and the dynamics of evidence accumulation for decision-making (B, C).

The proposed architecture consists of three main stages:

  1. Evidence Extraction: The neural network outputs an "evidence vector" at each time step rather than a direct policy.
  2. Accumulation: This evidence is summed over time in separate channels for each possible action.
  3. Confidence Thresholding: An action is executed only if the preference (calculated via softmax across channels) crosses a learned threshold ($\tau$).

This system was integrated with the Advantage Actor-Critic (A2C) algorithm and compared against a standard A2C-RNN baseline.

Experimental Results: The Mode Estimation Task

To evaluate the model, the authors developed the Mode Estimation (ME) Task. In this environment, an agent receives noisy samples from a discrete distribution and must identify the most frequent value (the mode). While waiting for more samples improves accuracy, it also reduces the final reward due to linear decay over time. Incorrect guesses or failing to guess within a time limit result in heavy penalties.

The task's difficulty is controlled by a stochasticity parameter ($\epsilon$), which determines the reliability of observations.

Performance metrics (Accuracy, Decision Time, Reward) across different levels of environmental noise ($\epsilon$).

Key Findings

  • Robustness to Uncertainty: As shown in Figure 5, the accumulator module maintains high accuracy and near-optimal rewards even as noise increases ($\epsilon = 0.6, 0.8$). In contrast, the A2C-RNN baseline's performance collapses under high uncertainty.
  • Natural Inaction: By making "no action" the default state, the accumulator module naturally handles ambiguity. If evidence is insufficient, preference values remain below the threshold, and the agent automatically waits for more information.
  • Sample Efficiency: Jointly training the evidence mapping and the confidence threshold proved more sample-efficient than traditional recurrent approaches, reaching solution stability in fewer iterations.

Conclusion and Future Work

The accumulator module provides a bridge between computational neuroscience and deep RL, offering a principled way for agents to handle uncertainty. While the Mode Estimation task demonstrates the core benefits, future work will focus on scaling this approach to high-dimensional visual inputs (e.g., Atari) and incorporating more complex biological mechanisms like the "hyperdirect pathway" for global action suppression in highly dynamic environments.

Summary updated

BibTeX
@misc{agarwal2018better,
  author = {Akshat Agarwal and Abhinau Kumar Venkataramanan and Kyle Dunovan and Erik Peterson and Timothy D. Verstynen and Katia P. Sycara},
  title = {{Better Safe than Sorry: Evidence Accumulation Allows for Safe Reinforcement Learning}},
  year = {2018},
  eprint = {1809.09147},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1809.09147},
}
Figure