Rendering-Free Event Frame Simulation for Fast Reinforcement Learning of Autonomous Robot Navigation

Dequan Ou, Austin Lopez, Quentin Missinne, Christophe De Wagter, Guido C. H. E. de CroonarXiv 2026 · arXiv

TL;DR

Researchers have developed an "analytic event proxy" that bypasses traditional visual rendering to simulate event camera data 400x faster than state-of-the-art methods. This breakthrough allows reinforcement learning (RL) policies for drone navigation to converge in under 30 minutes, enabling a 56g nano-drone to perform fully autonomous, event-driven obstacle avoidance using only a sub-watt onboard microcontroller.

What Problem Does This Paper Address?

Event cameras are a dream for agile robotics: they have microsecond latency, massive dynamic range, and consume mere milliwatts. This makes them perfect for tiny drones (nano-UAVs) that need to zip through cluttered rooms. However, training these drones using Reinforcement Learning is a computational nightmare.

Standard simulators follow a "render-then-convert" pipeline: they first generate a photorealistic image and then calculate brightness changes to synthesize events. This rendering step is incredibly slow, creating a bottleneck that makes large-scale RL training take days or even weeks. For a 56g drone with limited compute, we need a way to train robust policies quickly and efficiently.

Methods

The authors propose sidestepping the rendering process entirely with an analytic event proxy.

Instead of representing a scene with complex meshes and textures, they represent it as a collection of 3D edge points sampled on depth discontinuities (like the silhouettes of trees or the outlines of walls).

The rendering-free event-proxy pipeline: (a) Scene as 3D edge points, (b) Projection under ego-motion, (c) Firing mechanism, (d) Final event frame.

Key technical components include:

  • Direct Synthesis: Events are generated directly from scene geometry and the drone's ego-motion. If a point moves across the image plane, it "fires" events.
  • Saturating Trigger Model: To match real hardware (like the Prophesee GenX320), they use a calibrated model where pixel "mass" accumulates and saturates, mimicking how real sensors integrate events into frames.
  • End-to-End RL: They use Proximal Policy Optimization (PPO) to train a compact convolutional neural network (CNN) directly on these synthesized event frames.
  • Onboard Deployment: The final policy is quantized to an 8-bit integer (int8) format, occupying just 1.2 MB, and runs on an STM32N6 microcontroller mounted on a Bitcraze Crazyflie drone.

Datasets

The study doesn't rely on a static pre-recorded dataset but rather on procedural generation:

  • Training: 512 parallel environments with randomized obstacle densities and "edge-point" props.
  • Sim-to-Sim Validation: Policies were tested zero-shot in NVIDIA Isaac Sim using the EVIS plugin, which provides high-fidelity, ray-traced event simulation. This served as a "rendered-world holdout" to test if the proxy-trained policy could handle realistic visual data.
  • Real-World Evaluation: Three distinct obstacle courses of increasing difficulty (poles, boxes, plants, and tight passages).
  • Benchmarks: Results were compared against metrics from AvoidBench, a standard suite for obstacle avoidance.

Key Insights and Ideas

  • Geometry > Photometry: The core "aha moment" is that for event-based navigation, you don't need photorealism. Event cameras fundamentally measure moving contrast edges. By simulating only those edges, you capture the essential information needed for navigation while discarding 99% of the computational overhead.
  • The Simulation Bottleneck: In traditional pipelines, ~96% of the time per training step is spent rendering the frame. In this new approach, event synthesis is so fast that it's no longer the dominant cost; flight dynamics and learning updates take up the bulk of the time.
  • Zero-Shot Success: Despite being trained on a "sketch" of the world (points and lines), the policy transfers remarkably well to both high-fidelity simulations and the real world without any fine-tuning. This suggests that the "edge-only" representation is a robust abstraction for the task.

Key Results

  • Speedup: The simulator generates a 64×64 event frame in just 17 µs, which is over 400x faster than rendered pipelines.
  • Training Time: A full 40-million-step RL training run completes in just 34 minutes on a single RTX 4090. For comparison, the authors estimate the same training would take ~200 days using rendered events.
  • Performance: In unseen, cluttered Isaac Sim environments, the policy achieved an 83% success rate at an average velocity of 2.4 m/s.
  • Real-World Flight: Deployed on a 56g Crazyflie, the system achieved an 82% average success rate across three real-world arenas, running entirely onboard with no external processing or positioning.
The 56g flight platform featuring the GenX320 event camera and N6 microcontroller board.

Limitations and Future Work

While impressive, the authors note a few areas for growth:

  • Static Environments: The current proxy assumes a static world. Extending this to handle dynamic obstacles (like other drones or moving people) is a natural next step.
  • Texture Sensitivity: The policy's performance dropped slightly (from 83% to 70%) when moving to highly textured "sports hall" environments in simulation, indicating that while edges are primary, dense background texture can still introduce noise.
  • Control Complexity: The current work focuses on 2D-style navigation (fixed height). Future iterations could incorporate full 3D attitude control and faster decision rates to enable even more aggressive maneuvers.

Reference: Dequan Ou, et al. "Rendering-Free Event Frame Simulation for Fast Reinforcement Learning of Autonomous Robot Navigation." arXiv:2610.37543v1, Oct 2026.

Summary updated

Abstract

Event cameras offer low latency, high dynamic range, and milliwatt-level power consumption, making them ideal for agile robots such as tiny drones navigating through cluttered environments. However, existing simulators generally render photorealistic images before converting them into events, making large-scale reinforcement learning prohibitively slow. We propose an analytic event simulator that exploits what event cameras fundamentally measure - moving contrast edges - to synthesize event streams directly from scene geometry and ego-motion without visual rendering. Our simulator generates a 64x64 event observation in 17 microseconds on a single GPU - over 400x faster than a state-of-the-art rendering pipeline - enabling end-to-end reinforcement learning for goal-directed obstacle avoidance to converge in under 30 minutes - three orders of magnitude faster than training on rendered events. Policies transfer zero-shot to physically based event simulations (ray-traced Isaac Sim). Our policy reaches 83% success in cluttered courses unseen during training (mid-difficulty, traversability probability approx. 0.57), at 2.4 m/s average goal velocity, and retains 70% in a simulated texture-dense indoor hall. Furthermore, we quantized the policy to a 1.2 MB int8 network running fully onboard a 56-gram Crazyflie equipped with a Prophesee GenX320 event camera and an OpenMV N6 microcontroller board. Without any fine-tuning, external processing, or positioning, the policy achieves an average success rate of 82% (5/5, 5/6, 4/6 flights) across three real-world obstacle courses of increasing difficulty. Code: https://github.com/tudelft/rendering-free-event-nav
BibTeX
@misc{ou2026rendering,
  author = {Dequan Ou and Austin Lopez and Quentin Missinne and Christophe De Wagter and Guido C. H. E. de Croon},
  title = {{Rendering-Free Event Frame Simulation for Fast Reinforcement Learning of Autonomous Robot Navigation}},
  year = {2026},
  eprint = {2610.37543},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.37543},
}
Figure