Abstract: U-Net, known for its simple yet efficient architecture, is widely utilized for image processing tasks and is particularly suitable for deployment on neuromorphic chips. This paper introduces the novel concept of Spiking-UNet for image processing, which combines the power of Spiking Neural Networks (SNNs) with the U-Net architecture. To achieve an efficient Spiking-UNet, we face two primary challenges: ensuring high-fidelity information propagation through the network via spikes and formulating an effective training strategy. To address the issue of information loss, we introduce multi-threshold spiking neurons, which improve the efficiency of information transmission within the Spiking-UNet. For the training strategy, we adopt a conversion and fine-tuning pipeline that leverage pre-trained U-Net models. During the conversion process, significant variability in data distribution across different parts is observed when utilizing skip connections. Therefore, we propose a connection-wise normalization method to prevent inaccurate firing rates. Furthermore, we adopt a flow-based training method to fine-tune the converted models, reducing time steps while preserving performance. Experimental results show that, on image segmentation and denoising, our Spiking-UNet achieves comparable performance to its non-spiking counterpart, surpassing existing SNN methods. Compared with the converted Spiking-UNet without fine-tuning, our Spiking-UNet reduces inference time by approximately 90%. This research broadens the application scope of SNNs in image processing and is expected to inspire further exploration in the field of neuromorphic engineering.
Introduction
Spiking Neural Networks (SNNs) represent a promising frontier for low-power edge computing by mimicking the asynchronous, discrete communication of biological neurons. While SNNs have demonstrated success in classification tasks, their application to complex pixel-wise image processing—such as segmentation and denoising—has been hindered by challenges in information fidelity and training efficiency. This paper presents Spiking-UNet, a framework that adapts the widely-used U-Net architecture to the spiking domain, achieving high performance with significantly reduced energy consumption.
Challenges in Spiking Image Processing
Adapting U-Net to SNNs involves overcoming three major hurdles:
- Information Loss: Representing high-precision pixel values with binary spikes often leads to "spike vanishing" or saturation, destroying the fine-grained details needed for segmentation.
- Skip Connection Disparity: U-Net relies on concatenating features from the encoder and decoder. These features often exhibit vastly different data distributions, causing standard normalization techniques to fail during ANN-to-SNN conversion.
- Training Complexity: Training deep SNNs from scratch using Backpropagation Through Time (BPTT) is incredibly memory-intensive, often exceeding GPU limits for high-resolution image tasks.
The Spiking-UNet Framework
To address these issues, the researchers introduced several key innovations in the architecture and training pipeline.
Multi-Threshold Spiking Neurons
Traditional SNNs use a single firing threshold. Spiking-UNet introduces neurons with multiple thresholds (e.g., $V_{th,1}, V_{th,2}, V_{th,3}$). As shown in the figure above, this allows a neuron to fire multiple spikes of varying "weights" within a single time step. The authors mathematically determined that an optimal threshold series follows a power-of-two progression ($V_{th, i+1} = V_{th, i}/2$), effectively creating a bit-depth representation that preserves information fidelity even in short time windows.
Connection-wise Normalization
In standard U-Nets, skip connections merge features from different depths. Applying a single scale factor to these merged features often results in inaccurate firing rates. Spiking-UNet employs connection-wise normalization, which calculates independent scale factors for the encoder and decoder paths before concatenation. This ensures that features are properly aligned, maintaining a linear correlation between ANN activations and SNN spike rates.
Training via Spiking Flow
Instead of training from scratch, the framework utilizes a conversion and fine-tuning pipeline. A pre-trained ANN is converted to an SNN, which is then fine-tuned using Accumulated Spiking Flow (ASF-BP). This method optimizes the total "flow" of spikes over time, avoiding the memory overhead of tracking individual spikes at every micro-step.
Experimental Evaluation
The model was evaluated on benchmarks including retinal vessel segmentation (DRIVE), electron microscopy (EM), and high-resolution driving scenarios (CamSeq01), as well as standard denoising datasets (BSD).
- Performance: On the DRIVE dataset, Spiking-UNet achieved an F1 score of 0.794, nearly matching the original U-Net (0.806) and outperforming previous SNN-based fully convolutional networks.
- Efficiency: The fine-tuning strategy allowed the model to operate with only 10–20 time steps, a 90% reduction in inference latency compared to conversion-only methods.
- Energy Savings: On the CamSeq01 dataset, Spiking-UNet consumed approximately 1.94 mJ per inference—over 180 times less energy than its traditional ANN counterpart (353 mJ).
Conclusion
Spiking-UNet demonstrates that SNNs can achieve high-fidelity results in complex image processing tasks while maintaining the energy efficiency required for neuromorphic hardware. By addressing the fundamental issues of information loss and normalization in skip connections, this work paves the way for deploying advanced computer vision models on low-power edge devices.
Summary updated