Nonlinear Photonic Neuromorphic Chips for Spiking Reinforcement Learning

Shuiying Xiang, Yonghang Chen, Haowen Zhao, Shangxuan Shi, Xintao Zeng, Ya-Hui Zhang, Xingxing Guo, Yanan Han, Ye Tian, Yuechun Shi, Yue HaoarXiv 2025 · DOI · arXiv

TL;DR

Researchers have developed the first large-scale, fully functional photonic neuromorphic chip capable of performing both linear and nonlinear computations entirely in the optical domain. By deploying this hardware to run Spiking Reinforcement Learning (RL), they achieved ultra-low latency (320 ps) and high energy efficiency, matching the performance of traditional software-based algorithms on complex control tasks.

What Problem Does This Paper Address?

While photonic computing excels at high-speed linear operations (like matrix-vector multiplication), it usually hits a wall when it comes to nonlinear computations. In most current photonic neural networks, nonlinearities are handled in the digital domain. This requires frequent "opto-electronic" conversions—shuttling data between light and electricity—which introduces significant latency, consumes power, and prevents the creation of a truly "all-optical" neural network.

Furthermore, most photonic spiking neural networks (SNNs) have been limited to simple supervised learning tasks like image classification. This paper tackles the much harder challenge of Reinforcement Learning (RL), where an agent must learn to make decisions in dynamic environments. Training SNNs for RL is notoriously difficult due to the non-differentiable nature of "spikes" and the noise inherent in analog hardware.

Methods

The authors propose a 16-channel programmable incoherent photonic neuromorphic chip that integrates two distinct technologies:

  1. Linear Computation: A simplified 16×16 Mach-Zehnder Interferometer (MZI) mesh fabricated on a Silicon-on-Insulator (SOI) platform. This handles the synaptic weighting (Matrix-Vector Multiplication).
  2. Nonlinear Computation: A 16-channel Distributed Feedback laser with a Saturable Absorber (DFB-SA) array using III-V materials. This acts as the "neuron," mimicking the Leaky Integrate-and-Fire (LIF) behavior of biological neurons by only "firing" an optical spike when an input threshold is reached.

To make this hardware work with RL, they developed a Spiking Proximal Policy Optimization (PPO) architecture. Because hardware is never perfect, they used a three-step software-hardware collaborative training framework:

  • Software Pre-training: Training the model in a simulator using surrogate gradients.
  • In-situ Hardware Training: Using a Stochastic Parallel Gradient Descent (SPGD) algorithm to map weights directly onto the MZI's physical voltages.
  • Hardware-aware Fine-tuning: Re-training the software model to compensate for the specific noise and manufacturing variations of the physical chip.

Datasets

The architecture was evaluated using two classic Reinforcement Learning benchmarks from the OpenAI Gym:

  • CartPole: A discrete control task where the agent must balance a pole on a moving cart.
  • Pendulum: A continuous control task requiring the agent to swing a pendulum up and keep it upright.

Key Insights and Ideas

  • The "All-Optical" Layer: By co-designing the MZI mesh and the DFB-SA array, the researchers successfully deployed an entire layer of a spiking neural network in light, eliminating the need for intermediate digital processing.
  • Incoherent Advantage: Unlike many photonic chips that require perfectly phased "coherent" light, this system uses incoherent light. This makes it much more robust against phase instability and environmental noise.
  • Hybrid Material Strategy: The use of different materials (SOI for synapses and III-V for neurons) highlights a path forward for heterogeneous integration in photonic chips—using the best material for each specific job.
  • Closing the Reality Gap: The "hardware-aware fine-tuning" is a crucial "aha" moment. It acknowledges that analog hardware will always have imperfections and uses software to "learn" those quirks, resulting in a 98.5% inference accuracy.

Key Results

  • Ultra-Low Latency: The end-to-end latency for a full layer of the photonic SNN is just 320 picoseconds.
  • High Efficiency: The chip achieved 1.39 TOPS/W for linear computations and 987.65 GOPS/W for nonlinear computations.
  • Competitive Performance: In the CartPole task, the hardware converged to a reward of 200, and in the Pendulum task to -250. These results are comparable to traditional, power-hungry software-based PPO algorithms.
  • Scale: The system supports 272 trainable parameters, a significant step up for nonlinear photonic neuromorphic hardware.

Limitations and Future Work

  • Integration Level: While the linear and nonlinear components work together, they are currently separate modules. Future work will likely focus on monolithic integration (putting everything on a single chip) to further reduce size and loss.
  • Scaling Up: The current demonstration is 16 channels. While the authors suggest scaling to 150+ channels is theoretically possible, maintaining signal integrity and managing heat at that scale remains a challenge.
  • Multi-layer Depth: This study focuses on deploying a single layer on hardware. Real-world "Deep" RL will require cascading multiple photonic layers, which will require careful management of optical power losses between stages.

Summary updated

Abstract

Photonic computing chips have made significant progress in accelerating linear computations, but nonlinear computations are usually implemented in the digital domain, which introduces additional system latency and power consumption, and hinders the implementation of fully-functional photonic neural network chips. Here, we propose and fabricate a 16-channel programmable incoherent photonic neuromorphic computing chip by co-designing a simplified MZI mesh and distributed feedback lasers with saturable absorber array using different materials, enabling implementation of both linear and nonlinear spike computations in the optical domain. Furthermore, previous studies mainly focused on supervised learning and simple image classification tasks. Here, we propose a photonic spiking reinforcement learning (RL) architecture for the first time, and develop a software-hardware collaborative training-inference framework to address the challenge of training spiking RL models. We achieve large-scale, energy-efficient (photonic linear computation: 1.39 TOPS/W, photonic nonlinear computation: 987.65 GOPS/W) and low-latency (320 ps) end-to-end deployment of an entire layer of photonic spiking RL. Two RL benchmarks include the discrete CartPole task and the continuous Pendulum tasks are demonstrated experimentally based on spiking proximal policy optimization algorithm. The hardware-software collaborative computing reward value converges to 200 (-250) for the CartPole tasks, respectively, comparable to that of a traditional PPO algorithm. This experimental demonstration addresses the challenge of the absence of large-scale photonic nonlinear spike computation and spiking RL training difficulty, and presents a high-speed and low-latency photonic spiking RL solution with promising application prospects in fields such as real-time decision-making and control for robots and autonomous driving.
BibTeX
@misc{xiang2025nonlinear,
  author = {Shuiying Xiang and Yonghang Chen and Haowen Zhao and Shangxuan Shi and Xintao Zeng and Ya-Hui Zhang and Xingxing Guo and Yanan Han and Ye Tian and Yuechun Shi and Yue Hao},
  title = {{Nonlinear Photonic Neuromorphic Chips for Spiking Reinforcement Learning}},
  year = {2025},
  eprint = {2508.06962},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.06962},
  howpublished = {arXiv},
}
Figure