Déjà-Events: Multi-view 3D Geometry with Event Cameras via Sparsity-Aware Adaptive Gating

Arda Cinar Demirtas, Pablo Varela Bernal, Marc Pollefeys, Sunghwan HongarXiv 2026 · arXiv

TL;DR

Déjà-Events is a new event-only model for 3D reconstruction that predicts depth, camera pose, and pointmaps in a single forward pass. By distilling knowledge from a powerful RGB-based "teacher" model and introducing a novel Sparsity-Aware Adaptive Gating (SAAG) mechanism, it achieves state-of-the-art accuracy on real-world datasets while using only 10% of the parameters of previous leading methods.

What Problem Does This Paper Address?

Event cameras are revolutionary sensors that asynchronously report brightness changes at the microsecond level. Unlike traditional frame-based cameras, they don't suffer from motion blur or sensor saturation in high-speed or high-contrast scenes (like exiting a dark tunnel into bright sunlight).

However, building 3D reconstruction models for event cameras is difficult for two reasons:

  1. The Data Gap: We have massive, diverse datasets for RGB images with 3D labels, but event camera datasets with dense 3D ground truth are tiny and limited.
  2. Data Sparsity: Events only fire at edges and moving textures. Large parts of an event "image" are empty, which confuses standard transformer architectures designed for dense RGB grids.

Methods

The authors propose a "student-teacher" distillation approach. They take a frozen, pre-trained RGB model called Déjà View (DVLT) as the teacher. The student, Déjà-Events, is a copy of this architecture but sees event streams instead of images.

Key Technical Components:

  • Latent Distillation: Instead of training on 3D ground truth (depth/pose), the student is trained to reproduce the latent tokens (internal representations) that the teacher generates from aligned RGB frames. This allows the model to inherit the teacher's geometric "priors" without needing explicit 3D labels.
  • Parameter Efficiency: The authors use LoRA (Low-Rank Adaptation), training only about 1% of the model's parameters.
  • Sparsity-Aware Adaptive Gating (SAAG): This is the paper's core innovation. In the original transformer block, updates are scaled by a gate that is the same for every patch. SAAG introduces a per-patch correction based on event density. If a patch is empty, the model adjusts its refinement schedule accordingly, allowing the network to focus compute where the information actually is.
Overview of the distillation process and the SAAG-enhanced transformer blocks.

Datasets

  • EventScape: A simulated dataset (based on CARLA) used for the primary distillation training.
  • MVSEC: A real-world dataset of outdoor driving used for zero-shot evaluation.
  • DSEC: A high-resolution real-world driving dataset used for zero-shot evaluation.

Key Insights and Ideas

  • Inheriting Priors, Not Limitations: Because the student learns geometric features rather than just copying pixels, it can "see" structure in areas where the RGB teacher is completely blinded by glare. As shown in Figure 1, when a car exits a tunnel, the RGB teacher sees a white void and predicts a flat plane, while the event student correctly reconstructs the vehicles ahead.
  • Sparsity as a Feature: Rather than trying to make events look like images, SAAG treats the sparsity of events as a signal to steer the transformer's refinement process.
  • Zero-Shot Transfer: The model is trained only on simulated data but outperforms models trained on real data when tested on MVSEC and DSEC, proving the robustness of the distilled geometric priors.
Qualitative comparison at a tunnel exit: the event student maintains structure where the RGB teacher saturates.

Key Results

  • Superior Accuracy: On the DSEC dataset, Déjà-Events halves the error of the strongest baseline (0.15 vs. 0.30 Absolute Relative error).
  • Efficiency: The model achieves these results with roughly one-tenth the parameters of the previous state-of-the-art, EventVGGT (118M vs 1.16B parameters).
  • Refinement Scaling: Thanks to SAAG, the model's accuracy continues to improve as you increase the number of recursion steps ($K$) at inference time, whereas baseline models often degrade if pushed past their training range.
A full 3D pointmap reconstructed from events alone on the DSEC dataset.

Limitations and Future Work

  • Bounded by the Teacher: The student generally stays slightly behind the teacher's performance in well-lit conditions, as it is limited by the quality of the teacher's internal representations.
  • Temporal Resolution: Currently, the model sums events into bins, which discards some of the microsecond-level temporal information. The authors suggest that future work could involve reading these bins sequentially to provide geometry between frames rather than just at frame intervals.
  • Intrinsics Sensitivity: The model's 3D accuracy is sensitive to predicted camera intrinsics (like focal length). The authors note that using a pre-calibrated camera head could further reduce errors.

Summary updated

Abstract

Event cameras keep capturing structure where conventional cameras blur or saturate, which makes them attractive for 3D perception in fast or poorly lit scenes. Event datasets with dense 3D ground truth, however, are far smaller and less diverse than the image collections behind feed-forward 3D reconstruction models. We present Déjà-Events, an event-only model that predicts depth, camera pose and pointmaps from multiple views in a single forward pass. Rather than relying on scarce 3D labels, we distil it from Déjà View, a feed-forward RGB model that refines its tokens by repeatedly applying one shared transformer block. On paired RGB and event recordings, the event student learns to reproduce the frozen teacher's final tokens, which requires no depth or pose ground truth and trains only about 1% of its parameters. Because events are dense at edges and moving texture but nearly absent in textureless regions, we further introduce sparsity-aware adaptive gating, which adapts the shared block's gates to events and adjusts them per patch by event density, so that accuracy holds as refinement steps are added. Trained only on simulated data, Déjà-Events achieves the lowest pointmap error among the event-based methods we evaluate, on EventScape and zero-shot on the real MVSEC and DSEC datasets. On DSEC it halves the error of the strongest baseline (0.15 vs. 0.30 AbsRel) with about a tenth of its parameters. It also has the lowest depth error on EventScape and DSEC and the lowest trajectory error on EventScape.
BibTeX
@misc{demirtas2026deja,
  author = {Arda Cinar Demirtas and Pablo Varela Bernal and Marc Pollefeys and Sunghwan Hong},
  title = {{Déjà-Events: Multi-view 3D Geometry with Event Cameras via Sparsity-Aware Adaptive Gating}},
  year = {2026},
  eprint = {2611.21302},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21302},
}
Figure