This paper explores a fascinating intersection of neuroscience and robotics: Can a biologically plausible spiking neural network (SNN) control a full-size humanoid robot? While deep reinforcement learning (RL) has mastered humanoid walking in simulation, these models are far from how biological brains function. The authors bridge this gap by distilling a high-performance RL "expert" into a simulated 3D neural assembly, testing various learning rules to see which ones can actually keep a robot upright.
TL;DR
Researchers successfully trained a biologically plausible spiking neural network to control the Unitree H1-2 humanoid robot using "eligibility propagation" (e-prop). While traditional biological learning rules like global reward-modulated STDP failed the task, e-prop matched the performance of non-biological backpropagation, enabling the robot to achieve brief bipedal stepping in the real world.
What Problem Does This Paper Address?
In recent years, "brain organoids"—3D cultures of biological neurons—have been trained to perform basic computational tasks like speech recognition. This raises a provocative question: Could such a biological culture eventually control a complex machine like a humanoid robot?
Current humanoid control relies on Deep RL or Model Predictive Control, which are computationally heavy and biologically unrealistic. Conversely, most spiking neural networks used in robotics are either hand-tuned or trained using Backpropagation Through Time (BPTT), a method that is mathematically powerful but biologically impossible because it requires signals to travel backward through time. The authors set out to find a "biologically plausible" learning rule that is strong enough to handle the high-dimensional, unstable task of bipedal locomotion.
Methods
The researchers designed a simulated 3D neural assembly consisting of 512 Leaky Integrate-and-Fire (LIF) neurons, mimicking the structure of neurons grown on a micro-electrode array (MEA).
To train this network, they used Knowledge Distillation. First, they trained a standard RL "expert" policy to walk. Then, they used that expert to generate a dataset of "correct" actions, which the SNN attempted to mimic. They compared four distinct learning schemes, ranging from most to least biologically faithful:
- Fixed Reservoir: Recurrent weights are frozen; only the output "readout" layer is trained. This mimics current limitations in interacting with real in-vitro cultures.
- Reward-modulated STDP (R-STDP): A "three-factor" rule where synapses are updated based on local activity and a single, global reward signal (mimicking dopamine).
- Eligibility Propagation (e-prop): A biologically plausible approximation of BPTT. It uses local "eligibility traces" at each synapse and combines them with a per-neuron feedback signal.
- BPTT: The non-biological gold standard used as a performance baseline.
Datasets
The study utilized a dataset generated by a "Constraints as Terminations" (CaT) RL expert policy running in the MuJoCo physics engine. This dataset included a diverse range of velocity commands and directions to ensure the SNN learned a robust walking gait. Training was further refined using DAgger (Dataset Aggregation), an iterative process where the model's own mistakes are added back into the training set to prevent error accumulation.
Key Insights and Ideas
- The Necessity of Per-Neuron Feedback: The biggest "aha" moment was the failure of R-STDP. Because R-STDP uses a single global reward signal for the whole network, it lacks "spatial credit assignment"—it can't tell which specific neuron caused a fall. e-prop succeeded because it provides specific feedback to individual neurons, suggesting that biological brains likely possess a similar mechanism for targeted feedback.
- Reservoirs are Insufficient for Humanoids: A fixed 3D reservoir (where only the output is trained) reached a "performance plateau." It simply couldn't generate a rich enough variety of internal features to maintain the complex balance required for a 27-joint humanoid.
- Efficiency: The successful e-prop network operated at a mean firing rate of 10 Hz. This is highly efficient and matches the physiological observations of real neural cultures.
Key Results
- Simulation Success: In simulation, e-prop achieved a 100% survival rate, matching the non-biological BPTT baseline. In contrast, the fixed reservoir only managed ~44% survival, and R-STDP struggled at ~11%.
- Hardware Deployment: When deployed on the real Unitree H1-2 robot, the e-prop network produced genuine bipedal stepping for 3 to 9 seconds.
- The Reality Gap: While the simulation was perfect, the real-world robot eventually fell in every trial. The authors noted that while the SNN learned to balance, it was less accurate at tracking specific velocity commands compared to the RL expert.
Limitations and Future Work
The primary limitation was the sim-to-real gap. The RL expert was trained with "domain randomization" (varying friction, mass, etc.), but the distillation process into the SNN did not include these variations. This likely made the SNN too "brittle" for the messy physics of the real world.
Future work will focus on:
- Improving Velocity Tracking: Moving beyond just "staying upright" to precisely following directional commands.
- Domain Randomization: Incorporating more physical noise during the distillation process to improve hardware stability.
- Closing the Loop with Biology: The ultimate goal remains moving from a simulated 3D assembly to a physical one, potentially using real biological neurons on an MEA to control the robot.
Summary updated