Multi-compartment Neuron and Population Encoding improved Spiking Neural Network for Deep Distributional Reinforcement Learning

Yinqian Sun, Yi Zeng, Feifei Zhao, Zhuoya Zhao

Abstract

Inspired by the information processing with binary spikes in the brain, spiking neural networks (SNNs) exhibit significant low energy consumption and are more suitable for incorporating multi-scale biological characteristics. Spiking Neurons, as the basic information processing unit of SNNs, are often simplified in most SNNs which only consider LIF point neuron and do not take into account the multi-compartmental structural properties of biological neurons. This limits the computational and learning capabilities of SNNs. In this paper, we proposed a brain-inspired SNN-based deep distributional reinforcement learning algorithm with combination of bio-inspired multi-compartment neuron (MCN) model and population coding method. The proposed multi-compartment neuron built the structure and function of apical dendritic, basal dendritic, and somatic computing compartments to achieve the computational power close to that of biological neurons. Besides, we present an implicit fractional embedding method based on spiking neuron population encoding. We tested our model on Atari games, and the experiment results show that the performance of our model surpasses the vanilla ANN-based FQF model and ANN-SNN conversion method based Spiking-FQF models. The ablation experiments show that the proposed multi-compartment neural model and quantile fraction implicit population spike representation play an important role in realizing SNN-based deep distributional reinforcement learning.

Introduction

Spiking Neural Networks (SNNs) are often hailed as the "green" alternative to standard Deep Learning, utilizing binary spikes to process information in a manner similar to the human brain. However, most contemporary SNNs rely on the Leaky Integrate-and-Fire (LIF) model, which treats a neuron as a single point. In biological reality, neurons possess complex branching structures—specifically apical and basal dendrites—that perform sophisticated pre-processing before signals reach the cell body (soma).

This structural simplicity limits SNNs when tackling Distributional Reinforcement Learning (DRL). DRL requires an agent to represent a full probability distribution of potential rewards, necessitating the fusion of two distinct data streams: the current environment state and "quantile fractions" (mathematical markers for the distribution). Standard point neurons struggle to integrate these streams effectively, often leading to inferior performance on complex decision-making tasks compared to traditional Artificial Neural Networks (ANNs).

Methods: The MCS-FQF Architecture

The researchers introduce the Multi-compartment Spiking Fully Parameterized Quantile Function (MCS-FQF) model to bridge this gap. The architecture moves beyond simple point neurons to leverage biological structural complexity.

The overall architecture of the MCS-FQF model, showing the flow from environment state to action via multi-compartment neurons.

Core Innovations

  1. Multi-compartment Neuron (MCN) Model: Instead of a single node, each neuron features three functional zones. The basal dendrites receive state information from a Spiking CNN, while the apical dendrites receive quantile fraction information. The soma then integrates these inputs to determine whether to fire a spike.
  2. Population Encoding: To represent continuous numerical values (like fractions between 0 and 1) in a spiking format, the authors utilized a group of neurons with overlapping Gaussian receptive fields. This allows the network to represent precise values through collective activity rather than just the firing rate of a single neuron.
  3. Direct Training with STBP: Unlike many SNNs converted from pre-trained ANNs, MCS-FQF is trained from scratch using Spatio-Temporal Backpropagation (STBP). A "surrogate gradient" is used to handle the non-differentiable nature of the spike function.

Key Insights: Dendritic Integration and Temporal Dynamics

The central breakthrough of this work is the realization that dendrites are not merely passive wires but active filters. The apical dendrites can inhibit or boost signals from the basal dendrites, allowing a single spiking neuron to perform logic significantly more complex than a standard ReLU unit.

Furthermore, population encoding transforms simple scalars into high-dimensional spike patterns, making it easier for subsequent layers to distinguish subtle differences in reward distributions. This structural advantage is particularly evident in temporal tasks; on the Spiking Heidelberg Digits (SHD) dataset, the MCN achieved 89.39% accuracy, vastly outperforming the 63.11% achieved by standard LIF neurons.

Experimental Results

The model was evaluated on 19 Atari 2600 games via the Arcade Learning Environment, including complex titles like Hero and Video Pinball.

  • Superior Performance: MCS-FQF outperformed the vanilla ANN-based FQF baseline in 14 out of 19 games. In Video Pinball, it achieved a score nearly 1.7x higher than the traditional model.
  • Energy Efficiency: By replacing expensive floating-point multiplications with simple additions on neuromorphic hardware, MCS-FQF achieved a 3.2x reduction in normalized energy consumption compared to the ANN baseline.
  • Direct Training Advantage: The model significantly outperformed "Spiking-FQF" (the version created via ANN-to-SNN conversion), validating the approach of designing for spikes from the outset.

Discussion and Future Work

While MCS-FQF represents a significant step forward, the authors note several areas for improvement. The current model lacks the "back-propagating" spikes from the soma to the dendrites found in biological brains. Additionally, the added complexity of compartments creates challenges for gradient-based training, such as potential signal oscillations. Future research will explore automated methods, like genetic algorithms, to optimize the numerous biological parameters (conductance, leaky time constants) inherent in multi-compartment models.

Summary updated

Abstract

Inspired by the information processing with binary spikes in the brain, the spiking neural networks (SNNs) exhibit significant low energy consumption and are more suitable for incorporating multi-scale biological characteristics. Spiking Neurons, as the basic information processing unit of SNNs, are often simplified in most SNNs which only consider LIF point neuron and do not take into account the multi-compartmental structural properties of biological neurons. This limits the computational and learning capabilities of SNNs. In this paper, we proposed a brain-inspired SNN-based deep distributional reinforcement learning algorithm with combination of bio-inspired multi-compartment neuron (MCN) model and population coding method. The proposed multi-compartment neuron built the structure and function of apical dendritic, basal dendritic, and somatic computing compartments to achieve the computational power close to that of biological neurons. Besides, we present an implicit fractional embedding method based on spiking neuron population encoding. We tested our model on Atari games, and the experiment results show that the performance of our model surpasses the vanilla ANN-based FQF model and ANN-SNN conversion method based Spiking-FQF models. The ablation experiments show that the proposed multi-compartment neural model and quantile fraction implicit population spike representation play an important role in realizing SNN-based deep distributional reinforcement learning.
BibTeX
@article{sun2025multi,
  author = {Yinqian Sun and Yi Zeng and Feifei Zhao and Zhuoya Zhao},
  title = {{Multi-compartment Neuron and Population Encoding improved Spiking Neural Network for Deep Distributional Reinforcement Learning}},
  journal = {Neural Networks},
  year = {2025},
  doi = {10.1016/j.neunet.2024.106898},
}
Figure