4D Reconstruction with Events & RGB

Dulanga Weerakoon, Darshana Rathnayake, Van Minh Do, Hemanth Sabbella, Kaichen Zhou, Fangneng Zhan, Alok Prakash, Sanjay E. Sarma, Archan Misra, Paul Pu LiangarXiv 2026 · arXiv

TL;DR

This paper explores a "minimalist" approach to 4D reconstruction by augmenting the existing MonST3R model with event camera data. By simply concatenating RGB frames with event voxel grids, the researchers achieved a 3.4% average reduction in depth error, with the most significant performance gains occurring during high-speed object and camera motion where traditional RGB-only methods typically fail.

What Problem Does This Paper Address?

4D reconstruction—the process of capturing both the 3D geometry and the motion of a scene over time—is a holy grail for robotics and AR/VR. While state-of-the-art "feed-forward" models like MonST3R can estimate depth and pose directly from video, they are fundamentally limited by the hardware of standard RGB cameras.

Standard cameras capture frames at a fixed rate (e.g., 30 or 60 fps). When things move too fast, you get motion blur; when motion is non-uniform, you get temporal aliasing. Event cameras, by contrast, don't capture full frames. Instead, they report per-pixel brightness changes asynchronously at microsecond resolution. They are virtually immune to motion blur. The authors set out to see if this high-speed "event stream" could be integrated into existing RGB pipelines without requiring a total architectural overhaul.

Methods

The researchers chose a path of "minimal extension" to the MonST3R architecture.

  1. Input Representation: They convert the asynchronous event stream into a voxel grid with five temporal bins.
  2. Fusion: For each view, the 3-channel RGB image is concatenated with the 5-channel event voxel grid, creating an 8-channel input tensor.
  3. The Adapter: Since MonST3R’s pre-trained Vision Transformer (ViT) encoder expects 3 channels, they added a lightweight convolutional "adapter" layer to project the 8 channels back down to 3.
  4. Pipeline: These adapted features are passed through the standard MonST3R Siamese encoder and decoder. The system uses a sliding window approach to process pairs of frames/voxels, predicting pointmaps that are then aligned into a single coordinate system to form a dynamic point cloud.
Fig. 1: Overview of our event-augmented 4D reconstruction pipeline. Pairs of RGB frames and their corresponding event voxel grids are processed within a sliding window and passed through MonST3R to predict per-window pointmaps. Camera intrinsics and extrinsics estimated from overlapping windows align consecutive pointmaps into a single coordinate system, yielding a dynamic point cloud that captures scene geometry and motion over time.

Datasets

Because standard 4D reconstruction datasets don't typically include event data, the authors used the V2E simulator to generate synthetic event streams for existing benchmarks:

  • Sintel: A classic for optical flow and motion.
  • Bonn: Used for video depth estimation.
  • KITTI: A standard for autonomous driving/odometry.
  • PointOdyssey: A large-scale synthetic dataset for long-term point tracking.

The model was trained on augmented versions of PointOdyssey, TartanAir, and Spring.

Key Insights and Ideas

  • Complementary Modalities: The core "aha" moment is the confirmation that RGB and events play distinct, complementary roles. RGB provides the dense spatial information needed for depth, while events provide the precise motion cues needed to track dynamic scenes.
  • Minimalism is Effective: You don't necessarily need complex cross-attention mechanisms or Spiking Neural Networks to see immediate benefits from event data. Simple concatenation and a single-layer adapter were enough to outperform the RGB-only baseline.
  • The "Motion Gap": The study reveals that the benefit of event data isn't constant; it scales with the difficulty of the scene. As object motion increases, the performance of RGB-only models drops sharply, while the event-augmented model remains much more robust.

Key Results

The event-augmented model, referred to as RGB + Ev. Vox., consistently outperformed the baseline across multiple metrics:

  • Depth Accuracy: On average, the fusion model reduced Absolute Relative (Abs Rel) error by 3.4%.
  • Pose Estimation: On the Sintel dataset, the fusion model achieved the lowest Mean Absolute Trajectory Error (ATE) and Relative Pose Error (RPE) compared to RGB-only or event-only variants.
  • Sensitivity to Motion: As shown in the sensitivity analysis, the gap between RGB and RGB+Events widens significantly in "High" motion categories. For high object motion, the fusion model achieved a 0.327 lower Log RMSE than the RGB-only version.
Fig. 3: Sensitivity of reconstruction error to camera and object motion magnitude on a subset of the PointOdyssey test set with MonST3R. RGB + Ev. Vox. consistently achieves lower Log RMSE than RGB alone, with the gap widening under high object motion (Figure 3b). Camera motion is divided into three levels (Low, Medium, High) based on the ground-truth camera poses, and object motion is divided into two levels (Low, High) based on the ground-truth depth variation.

Qualitatively, the reconstructions maintain scene geometry even in complex, dynamic sequences where standard methods might produce warped or blurred results.

Qualitative 4D reconstructions produced by the fusion model

Limitations and Future Work

While promising, the authors acknowledge several areas for improvement:

  • Synthetic Events: The study relies on simulated event data (V2E). Real-world event cameras have noise characteristics and latencies that simulators may not perfectly capture. Validating on real event-camera hardware is a critical next step.
  • Simple Fusion: Concatenation treats RGB and events equally everywhere. Future work could use adaptive fusion (like gating or cross-attention) to let the model "trust" events more in high-motion areas and trust RGB more in static, well-lit areas.
  • Encoder Efficiency: Using a standard ViT for sparse event data is somewhat inefficient. Exploring event-specific backbones like Spiking Neural Networks (SNNs) or Event Transformers could yield better results with lower computational overhead.

Summary updated

Abstract

Reconstructing dynamic 3D scenes over time (4D reconstruction) from a single moving camera is central to robotics and AR/VR applications, yet state-of the-art feed-forward methods rely solely on RGB frames and therefore inherit the failure modes of conventional cameras, including motion blur and temporal aliasing under fast or non-uniform camera and object motion. Event cameras report per-pixel brightness changes asynchronously at microsecond resolution, offering a complementary signal that is largely immune to these artifacts, but whether this modality can be incorporated into existing RGB based 4D reconstruction pipelines without a full architectural redesign remains an open question. In this preliminary study, we investigate a minimal extension: we augment Monst3R, an existing RGB-only 4D reconstruction model, with an additional event-camera input branch, leaving the rest of the architecture unchanged. Across 3 benchmark datasets, this simple RGB-event fusion reduces Abs Rel by 3.4% on average relative to the RGB-only Monst3R baseline, and a sensitivity analysis shows that the improvement grows with the magnitude of camera and object motion, with the largest gains observed under high object motion conditions where RGB-only reconstruction degrades most. These early results suggest that even a lightweight, non-invasive fusion of event data can meaningfully improve 4D reconstruction robustness, motivating a more thorough architectural investigation in future work.
BibTeX
@misc{weerakoon2026d,
  author = {Dulanga Weerakoon and Darshana Rathnayake and Van Minh Do and Hemanth Sabbella and Kaichen Zhou and Fangneng Zhan and Alok Prakash and Sanjay E. Sarma and Archan Misra and Paul Pu Liang},
  title = {{4D Reconstruction with Events \& RGB}},
  year = {2026},
  eprint = {2610.36610},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36610},
}
Figure