Spiking Event-Camera Receivers That See Optical Blockages Coming

Light-based wireless links, such as LED lighting that carries data or lets a device locate itself, have one hard weakness. They need a clear line of sight. When a person walks between the lamp and the receiver, the signal does not fade gradually. It drops to roughly zero almost instantly On Optimizing VLC Networks for Downlink Multi-User Transmission: A Survey. The blocker is visible before it arrives, though. An event camera is a sensor whose pixels report only brightness changes, each with a microsecond timestamp. It sees the occluder's moving edge approach the lamp before the edge covers it. We propose a receiver that decodes the LED beacons and also forecasts, per lamp, when each link is about to be cut. The receiver is built from spiking neural networks: networks of neurons that communicate with sparse binary pulses, so computation and energy scale with how much is happening. With the forecast, the device can switch to other lamps, or warn a reflector controller, before the dropout instead of after it.

The design addresses a tension that the current pairing of event cameras and spiking networks ignores. A flickering LED is the worst input for a sparsity-based processor. Every on/off transition produces a burst of events, and that flood erases the energy advantage that spiking hardware depends on. Our receiver therefore splits the event stream in two. Carrier events are compressed by frequency-tuned spiking units into roughly one pulse per beacon period. Only the leftover non-periodic events, which come from motion and occluder edges, reach a small learned spiking forecaster. The aim is a receiver whose energy tracks how much is changing in the scene rather than how fast the lamps blink.

Core Claim

Our main claim has two parts. First, imminent line-of-sight loss can be predicted from the occluder-edge events that surround each beacon in an event stream, without channel state information (per-link measurements of signal gain), with useful lead time of tens to hundreds of milliseconds. Second, a two-stage spiking receiver can do this jointly with beacon decoding and localization, while its operation count stays close to that of a decoder with no forecasting.

The first stage is a bank of resonate-and-fire neurons. These are spiking units whose potential oscillates, so they fire strongly only when input arrives at their natural frequency. We borrow the neuron model from A Spiking Heterogeneous Harmonic Resonate-and-Fire State Space Model for Time Series. That paper uses such neurons for generic time-series modeling. We repurpose them as per-pixel beacon identifiers. The second stage is a recurrent leaky integrate-and-fire network trained with surrogate gradients. It reads the non-carrier residual events in a ring around each detected beacon and outputs a per-beacon time-to-blockage. It also outputs a continuous position estimate through a non-firing accumulator readout, following the regression technique of Spiking neural networks for frame-based and event-based single object localization.

Positioning Against Prior Work

Two papers motivate this proposal by showing that event cameras are the right sensor for optical links. E-VLC decodes flickering LED beacons with a 1-Mpixel event camera and estimates 6-DOF pose. It reaches 14 m range against 4–5 m for ArUco markers, keeps over 90% detection at about 600 lux, and reports mean localization errors of 17–25 cm under severe rotation. Motion-Aware Optical Camera Communication with Event Cameras keeps nearly 100% tracking success under motion, against about 85% for QR codes. Both papers decode with dense conventional algorithms: contrast maximization plus a pose solver in the first, contour extraction and Reed-Solomon decoding in the second. Neither predicts blockage. We keep their sensor and their evaluation data, but replace the processing and add forecasting.

Two papers motivate the forecasting goal. In radio vehicular networks, Dynamic Blockage Pre-Avoidance using Reconfigurable Intelligent Surfaces shows that anticipating blockers recovers 50% of throughput loss, using large-scale trajectory reports rather than per-element channel measurements. Our receiver supplies an analogous CSI-free blockage signal for optical links, computed locally from the sensor. High-Speed Robust Dynamic Positioning and Tracking tracks through 50–100% shielding using frame-based optical flow, Bayesian forecasting and Kalman filtering. It reaches 0.86 cm error at speeds up to 48 km/h, with 0.162 s per cycle. It coasts through occlusions once they happen; it does not predict their onset.

Several papers constrain the design. The material's own tension, that continuous modulation floods event-driven processors, comes from the need for the noise-suppressing thresholds in Bio-Vision-Inspired Spiking Neural Networks for Object Detection with Event Cameras. That paper's 262× energy claim was measured in GPU simulation, not on neuromorphic chips. So we treat simulated energy as provisional and seek on-chip measurement, as Reinforcement co-Learning of Deep and Spiking Neural Networks did on Loihi: 15.53 µJ against 1171.70 µJ on a Jetson TX2. The reflector-control works Deep Q-Learning-Based Resource Management in IRS-Assisted VLC Systems and Placement, Orientation, and Resource Allocation Optimization for Cell-Free OIRS-aided OWC Network assume perfect channel state information and static users. Mirror arrays are mechanical, which Metasurfaces vs Mirror Arrays implies by favoring them for power focusing. For both reasons, advance warning is worth more to such a reflector controller than faster computation. We do not build that controller; we produce the input it lacks.

Research Program

Training data. We will extend the E-VLC sequences with controlled occlusions. One part is synthetic: occluders of human, hand and door shapes, with known trajectories, rendered over the scenes and converted to events with ESIM. The other part is real: a small recorded set of people walking between ceiling LEDs and a handheld or robot-mounted event camera. Labels for when each beacon disappears come from the beacon-decoding ground truth, giving a per-beacon time-to-loss target.

Stage one. We will study whether resonate-and-fire banks, fixed or with learned frequencies, can separate carrier events from residual events reliably under fast camera rotation. This is the regime where E-VLC needed motion compensation. We will also measure how much of the raw event rate they remove.

Stage two. We will train the forecaster and the position readout jointly. The forecaster's outputs drive a simple policy: drop a beacon from the pose solve, or hand over to another lamp, once its predicted time-to-loss falls below a threshold.

Energy accounting. We will count accumulate-only operations and spikes for the spiking stages. For the dense alternatives we will count multiply-accumulates. If Loihi access is available, we will port the forecaster and report measured per-inference energy.

Baselines

Naive reference. E-VLC-style decoding with no anticipation. Blockage is detected reactively when a beacon's events stop, and pose is coasted with a Kalman filter. Every forecasting gain is measured against this. Beating it means positive median lead time at matched false-alarm rate, and a smaller fraction of time with degraded pose.

Strongest existing pipeline. The E-VLC contrast-maximization plus pose-solver pipeline E-VLC, run on the same sequences. Its reported mean localization error is 17–25 cm. No reported result was found on Lacuna for its energy or operation count, so we will measure it ourselves on the same CPU. To count as matching it, our receiver must stay within that error band on unoccluded E-VLC sequences while using markedly fewer operations.

Frame-based forecast-and-coast tracker. The optical-flow plus Bayesian tracker High-Speed Robust Dynamic Positioning, re-run on frames synchronized with the events. E-VLC provides 88,000 frames. Its reported figures are 0.86 cm on its own rig and 0.162 s per cycle. Those figures are not directly comparable to E-VLC scenes, so the comparison is on our occlusion set. We beat it if we warn about blockage before it occurs, which it does not attempt, while staying under its per-cycle latency.

Matched dense network. The same two-stage architecture, implemented as a dense recurrent convolutional network on event voxel grids. It gets the same inputs, the same training data and the same hyperparameter search budget. This separates the effect of spiking computation from the effect of the decomposition itself.

Ablations. Two ablations isolate the key ingredient. The first feeds all events, carrier included, directly to the spiking forecaster. We expect forecasting accuracy to be similar but spike counts to rise sharply, which would confirm that the carrier filter is what preserves sparsity. The second removes the ring of residual events around each beacon, to test whether edge events really carry the predictive signal.

Why not just…?

Why not just use high-speed CMOS cameras with fiducial markers? The evidence already in hand says frame sensors lose on range, motion and low light. ArUco fails beyond 4–5 m against 14 m for events, and loses detection at about 600 lux E-VLC. Frame-based tracking under motion drops to about 85% success Motion-Aware OCC. A frame camera could still anticipate occluders, so the frame-based tracker baseline is run on our occlusion set to test that directly.

Why not just run a small dense network on an embedded GPU? This is a real threat on accuracy. If the matched dense network forecasts equally well, the case rests entirely on energy. That case only matters for always-on, battery-powered receivers such as wearables and small robots; it does not matter for mains-powered infrastructure. The existing hardware evidence favors spiking networks, with a 75× gain on Loihi in Reinforcement co-Learning. The material also shows that event floods can cancel this gain. Our full-event ablation and the operation counts will settle the question, and we will report a loss if the dense network wins on total cost.

Why not just decode beacons with a hand-written per-pixel timing detector and a Kalman filter? For decoding alone, a hand-written interval detector may well match resonate-and-fire neurons at lower cost. We test this head-to-head as a drop-in replacement for stage one. If it wins, we adopt it, and the contribution narrows to the forecaster. A Kalman filter extrapolates the receiver's own motion. It cannot see a third party walking toward the lamp, so it cannot provide lead time. The naive baseline quantifies that gap.

Why not just re-steer reflecting surfaces faster? The reflectors that focus light best are mechanical mirror arrays Metasurfaces vs Mirror Arrays. Their response time is bounded by actuator movement, not by computation. Faster computation therefore helps less than early warning, which is what Dynamic Blockage Pre-Avoidance exploits in the radio setting.

Proposed Experiments

Decisive check. On held-out real occlusion recordings, the forecaster must reach a median lead time long enough to cover a lamp handover. Precision and recall are reported at a fixed false-alarm rate per minute. The fraction of time with pose error above 25 cm must fall relative to the naive baseline.

Secondary checks. First, unoccluded E-VLC localization and beacon detection must remain within the reported 17–25 cm and above 90% ranges, including the low-light sequences. Second, we report spikes and operations per second as a function of the number of visible beacons and the amount of scene motion. Our claim predicts that operations grow with motion but stay nearly flat as beacon flicker frequency increases. Third, we report on-chip energy for the forecaster where hardware allows, with simulation-only numbers labeled as such.

Failure case. E-VLC notes that reflective surfaces create ghost beacons. We will test whether they also trigger false blockage alarms. This is the failure we consider most likely.