Reading Redirection Stress from the Eye: Event-Driven Per-User Gain Control for Redirected Walking

Redirected walking lets a person walk naturally through a large virtual world inside a small room. The system slowly rotates or rescales the virtual camera so the user drifts along a path that stays inside the room without noticing. How much drift a person tolerates differs widely between people. Today every user gets the same conservative population threshold, because measuring an individual's threshold takes hundreds of explicit "did the world just rotate?" trials. Recent work shows the body gives this away without being asked: gaze velocity and postural sway rise measurably with redirection strength even when users do not consciously detect it (Sensitivity to Redirected Walking Considering Gaze, Posture, and Luminance). We propose to close the loop. A small always-on decoder turns those involuntary signals into a running estimate of detection risk, and the redirection controller uses it to push each user's gains up to their own limit and back off before they notice.

The neuromorphic side enters as a testable hypothesis about where the signal lives and what it costs to read. When the camera is rotated faster than the head, the eye's reflexive stabilization no longer matches the scene, so the image slips across the retina and the eye makes fast corrective movements. These events last milliseconds and are sparse. Event cameras are sensors that report only per-pixel brightness changes, asynchronously, and spiking neural networks are networks that compute only when such events arrive. Together they suit a signal that is brief, rare, and needs watching all the time on a battery-powered headset. Our core claim has two parts. Per-user, physiology-driven gains reduce resets relative to population-fixed gains without raising conscious detection. And high-temporal-resolution event sensing of the eye recovers detection information that frame-based eye tracking at headset rates aliases away, at a power cost low enough to run continuously. The second part can fail independently of the first, and the design reports either outcome.

Why this, and not a spiking redirection controller

The obvious meeting point of these two fields is to train a spiking reinforcement-learning policy that outputs redirection gains and run it on neuromorphic silicon. Spiking actors for continuous control already exist: population-coded actors match deep actors on MuJoCo locomotion at 140x less energy per inference on Loihi (Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control), and a hybrid spiking actor with a deep critic navigates at 15.53 µJ per inference on Loihi versus 1171.70 µJ for DDPG on a Jetson TX2 (Reinforcement co-Learning of Deep and Spiking Neural Networks for Energy-Efficient Mapless Navigation with Neuromorphic Hardware). We do not pursue this, for two reasons.

The first reason is energy. Even the dense number, about 1.2 mJ at 90 Hz, works out to roughly 0.1 W. That is a small slice of a standalone headset's budget. The strongest geometric controllers are also reactive and cheap per frame: the alignment-based controller ARC samples three obstacle distances (ARC: Alignment-based Redirection Controller for Redirected Walking in Complex Environments). A skeptic would rightly say "just run ARC on the CPU."

The second reason concerns where the gains are. In a real shared room, the gap between no redirection and the best predictive controller is small: 5.28 m versus 5.75 m mean distance between resets (Predictive multiuser redirected walking using artificial potential fields). This suggests that smarter steering geometry has limited headroom while gains stay capped at conservative thresholds. Raising the cap per user is the less-explored lever. Reading the physiology continuously, cheaply, and fast is the step where event-driven computation has a real case.

Core Mechanism

The system has three parts. The first is a sensing front end: an event sensor on each eye plus the headset's existing head-pose stream, from which sway is computed. The second is a small spiking decoder that reads these streams. The third is a gain governor that wraps an existing geometric steering controller.

The decoder encodes continuous inputs such as head displacement and pupil position with populations of neurons with Gaussian tuning, following the population-coding approach of PopSAN. It encodes the eye's event stream directly as spikes, and it feeds a single output "risk" neuron whose membrane potential acts as a leaky evidence accumulator:

$$V_{t+1} = \lambda V_t + \sum_i w_i s_{i,t}, \qquad \text{risk spike when } V_t > \theta,$$

where $V_t$ is the risk neuron's potential at step $t$, $\lambda \in (0,1)$ is the leak that sets how long evidence persists, $s_{i,t} \in \{0,1\}$ are input spikes, $w_i$ are learned weights, and $\theta$ is the firing threshold. The leak matters for the design: it encodes the assumption that detection risk builds over seconds of sustained mismatch rather than from single glitches. We will fit $\lambda$ rather than fix it. The network is trained offline with surrogate gradients on labeled trials, where the label is the user's 2-alternative-forced-choice (2AFC) response and the regression target is the applied gain. It is then deployed on Speck or Loihi.

The governor multiplies the steering controller's commanded gains by a per-user margin $m$:

$$m \leftarrow m + \alpha\,(1 - r) - \beta\, r,$$

where $r$ is 1 if the risk neuron spiked in the last window and 0 otherwise, $\alpha$ is a slow upward step, and $\beta \gg \alpha$ is a fast backoff. This is a staircase procedure, the classic psychophysics method, with the user's explicit answer replaced by the involuntary signal. The steering direction still comes from ARC or from visibility-polygon steering (Redirected Walking in Static and Dynamic Scenes Using Visibility Polygons). Only the magnitude cap is personalized. This keeps the contribution orthogonal to controller geometry.

Research Program

The work has three phases.

Phase one: does the signal exist at high temporal resolution? We replicate the rotation-trial protocol of the sensitivity study (90° rotations, Meta Quest Pro, photopic and mesopic lighting) with event sensors added beside the built-in eye cameras. Each trial is recorded simultaneously by the event sensor and the frame tracker. That study fit multilevel regressions offline and reported effects of 5.096 cm of sway and 0.9034°/s of gaze velocity per unit of rotation gain. It reported no per-trial detection accuracy, so no reported result was found on Lacuna for per-trial detection AUC from physiology. We will establish that number ourselves.

Phase two: does closing the loop reduce resets in simulation? We fit a generative model of each Phase one participant's physiological response as a function of gain and their personal threshold. We then use it to drive simulated users in established redirected-walking environments.

Phase three: does it hold up with real walkers? We run a within-subject user study in a real tracking space and measure on-chip energy.

Baselines

The naive reference is the steering controller run at fixed population-threshold gains with no personalization. This is how ARC and visibility-polygon steering are deployed today. All personalized conditions share its steering directions, reset policy, and environments, so any difference comes from the gain cap alone.

The strongest existing method for setting a personal cap is explicit staircase calibration. The user answers 2AFC questions before the session, and their threshold is fixed from those answers. It is the accuracy ceiling for a one-shot estimate, at the cost of minutes of calibration. We give it the same number of pre-session minutes our decoder's per-user fine-tuning uses, and we beat it if online physiological gains match or exceed its reset reduction with zero explicit trials and equal or lower post-session detection.

The strongest competitor to the neuromorphic part is a dense decoder on the headset's existing frame-based eye tracker and head pose. It is a conventional network of matched parameter count and the same training data and hyperparameter search budget, running on the headset CPU. If it matches our decoder's detection AUC and closed-loop reset counts, the event-sensing claim fails. The energy comparison is then reported as a deployment detail, not a contribution.

Two ablations isolate the key ingredients. A non-spiking leaky integrator over the same event features, run on CPU, separates the value of event sensing from the value of spiking hardware. A head-sway-only decoder uses signals the headset already tracks for free and tells us whether eye sensing is needed at all.

Why not just…?

Why not just read gaze velocity from the eye tracker the headset already has? This is the most serious threat. If the detection signal sits in slow average gaze velocity, as the sensitivity study's regression framing suggests, a 90–120 Hz tracker should capture it. The proposal settles this directly. Phase one decodes the same trials from the event stream and from frame streams downsampled to 30, 90, and 200 Hz, and plots AUC against bandwidth. A flat curve means the frame-based decoder wins on simplicity, and we would drop the event sensor while keeping the closed loop.

Why not just improve the steering geometry instead? Visibility-polygon steering cut median resets to 121 versus 153 for ARC in corridors and to about 87 versus over 150 for potential fields in clutter (Redirected Walking in Static and Dynamic Scenes Using Visibility Polygons). Optimized reset directions remove a further 30% to over 70% of resets (Adaptive Optimization Algorithm for Resetting Techniques in Obstacle-Ridden Environments). These are complementary, not competing. We wrap the best of them and test whether personalization adds a reduction on top. A multiplicative improvement over visibility-polygon steering alone is the result that matters.

Why not just calibrate once with a short staircase? If a user's threshold is stable within a session, a single short calibration captures most of the benefit and online decoding is unnecessary. Phase one measures within-session threshold drift by interleaving staircase blocks early and late in long sessions. Phase three includes the staircase baseline. If drift is negligible and the staircase matches us, the honest conclusion is that zero-trial calibration is the contribution, not continuous adaptation.

Why not just convert a trained network to a spiking one after the fact? Conversion tools such as burst-ann2snn are a reasonable deployment route. Prior navigation work found direct conversion 1% to 4.2% less successful than hybrid co-learning (Reinforcement co-Learning of Deep and Spiking Neural Networks for Energy-Efficient Mapless Navigation with Neuromorphic Hardware). We will report both, since this choice is secondary to the main claims.

Proposed Experiments and Decisive Checks

Phase one recruits a larger cohort than the 11 participants of the original sensitivity study, with thousands of rotation trials. It reports four things: per-trial detection AUC under event and frame sensing at each bandwidth, the fitted leak timescale, the luminance effect (the original found none, p = 0.238, and our event sensor should be similarly insensitive), and within-session drift.

Phase two runs the simulated users over 100 random collision-free virtual paths per condition in ARC's environments A, B, and C. It also runs them in the visibility-polygon corridor and clutter suites, with personal thresholds drawn from the Phase one distribution. The metrics are resets, distance between resets, and the fraction of time each simulated user's gains exceed their true threshold. That last quantity is a proxy for the detection rate, available because the simulator knows each user's ground truth. The decisive check is a reset reduction over fixed-gain visibility-polygon steering with no increase in that above-threshold fraction.

Phase three is a counterbalanced within-subject study in a cluttered tracking space. It compares fixed gains, staircase-calibrated gains, the frame-based decoder, and the event-driven spiking decoder. The outcomes are resets, distance between resets, post-session detection reports, and simulator sickness and presence questionnaires. The reference effect size is the 5.28 to 5.75 m spread among controllers in the multiuser study. A personalization gain of that size or larger would show the gain cap is at least as important a lever as steering geometry.

Energy is measured on Speck and Loihi against the dense decoder on a mobile CPU, separately for sensor and compute. We report end-to-end latency against the 2.3 ms figure achieved by the event pipeline in Fully Autonomous Neuromorphic Navigation and Dynamic Obstacle Avoidance, whose codebase we reuse for the event-processing stack.

The proposal rests on two decisive checks. First, does event bandwidth raise detection AUC over 90 Hz frames? Second, does physiological personalization reduce resets on top of the best geometric controller without raising detection? A yes to the second and a no to the first still yields a useful, simpler system. A no to the second ends the line.