TL;DR
Déjà-Events is a new event-only model for 3D reconstruction that predicts depth, camera pose, and pointmaps in a single forward pass. By distilling knowledge from a powerful RGB-based "teacher" model and introducing a novel Sparsity-Aware Adaptive Gating (SAAG) mechanism, it achieves state-of-the-art accuracy on real-world datasets while using only 10% of the parameters of previous leading methods.
What Problem Does This Paper Address?
Event cameras are revolutionary sensors that asynchronously report brightness changes at the microsecond level. Unlike traditional frame-based cameras, they don't suffer from motion blur or sensor saturation in high-speed or high-contrast scenes (like exiting a dark tunnel into bright sunlight).
However, building 3D reconstruction models for event cameras is difficult for two reasons:
- The Data Gap: We have massive, diverse datasets for RGB images with 3D labels, but event camera datasets with dense 3D ground truth are tiny and limited.
- Data Sparsity: Events only fire at edges and moving textures. Large parts of an event "image" are empty, which confuses standard transformer architectures designed for dense RGB grids.
Methods
The authors propose a "student-teacher" distillation approach. They take a frozen, pre-trained RGB model called Déjà View (DVLT) as the teacher. The student, Déjà-Events, is a copy of this architecture but sees event streams instead of images.
Key Technical Components:
- Latent Distillation: Instead of training on 3D ground truth (depth/pose), the student is trained to reproduce the latent tokens (internal representations) that the teacher generates from aligned RGB frames. This allows the model to inherit the teacher's geometric "priors" without needing explicit 3D labels.
- Parameter Efficiency: The authors use LoRA (Low-Rank Adaptation), training only about 1% of the model's parameters.
- Sparsity-Aware Adaptive Gating (SAAG): This is the paper's core innovation. In the original transformer block, updates are scaled by a gate that is the same for every patch. SAAG introduces a per-patch correction based on event density. If a patch is empty, the model adjusts its refinement schedule accordingly, allowing the network to focus compute where the information actually is.
Datasets
- EventScape: A simulated dataset (based on CARLA) used for the primary distillation training.
- MVSEC: A real-world dataset of outdoor driving used for zero-shot evaluation.
- DSEC: A high-resolution real-world driving dataset used for zero-shot evaluation.
Key Insights and Ideas
- Inheriting Priors, Not Limitations: Because the student learns geometric features rather than just copying pixels, it can "see" structure in areas where the RGB teacher is completely blinded by glare. As shown in Figure 1, when a car exits a tunnel, the RGB teacher sees a white void and predicts a flat plane, while the event student correctly reconstructs the vehicles ahead.
- Sparsity as a Feature: Rather than trying to make events look like images, SAAG treats the sparsity of events as a signal to steer the transformer's refinement process.
- Zero-Shot Transfer: The model is trained only on simulated data but outperforms models trained on real data when tested on MVSEC and DSEC, proving the robustness of the distilled geometric priors.
Key Results
- Superior Accuracy: On the DSEC dataset, Déjà-Events halves the error of the strongest baseline (0.15 vs. 0.30 Absolute Relative error).
- Efficiency: The model achieves these results with roughly one-tenth the parameters of the previous state-of-the-art, EventVGGT (118M vs 1.16B parameters).
- Refinement Scaling: Thanks to SAAG, the model's accuracy continues to improve as you increase the number of recursion steps ($K$) at inference time, whereas baseline models often degrade if pushed past their training range.
Limitations and Future Work
- Bounded by the Teacher: The student generally stays slightly behind the teacher's performance in well-lit conditions, as it is limited by the quality of the teacher's internal representations.
- Temporal Resolution: Currently, the model sums events into bins, which discards some of the microsecond-level temporal information. The authors suggest that future work could involve reading these bins sequentially to provide geometry between frames rather than just at frame intervals.
- Intrinsics Sensitivity: The model's 3D accuracy is sensitive to predicted camera intrinsics (like focal length). The authors note that using a pre-calibrated camera head could further reduce errors.
Summary updated