CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecasting

Jiezhi Yang, Khushi Desai, Charles Packer, Harshil Bhatia, Nicholas Rhinehart, Rowan McAllister, Joseph GonzalezarXiv 2024 · DOI · OpenReview

TL;DR

This paper introduces CARFF, a novel method for predicting future 3D scenes from partial 2D observations, crucial for autonomous driving in uncertain environments. CARFF uses a conditional auto-encoder and Neural Radiance Fields (NeRF) to generate probabilistic 3D scene forecasts, enabling robust planning even with occlusions.

What Problem Does This Paper Address?

Autonomous systems, particularly in driving, often operate with incomplete information due to occlusions. Traditional methods often simplify sensor data into low-dimensional states, losing crucial information for robust decision-making. Humans, however, can infer unseen elements and reason about potential future states based on partial observations and prior knowledge. This paper addresses the challenge of creating AI systems that can achieve similar probabilistic reasoning about unseen parts of a 3D environment, especially for forecasting future scene evolution and enabling effective planning under uncertainty. This is framed as a Partially Observable Markov Decision Process (POMDP) problem.

Methods

CARFF employs a two-stage training process:

  1. Pose-Conditional VAE (PC-VAE): This component takes 2D ego-centric images and their camera poses as input. It learns to encode these into pose-invariant latent scene representations. The key innovation here is the "pose-conditional" aspect, which helps disentangle scene content from camera viewpoint, making the latents more robust and 3D-aware. The VAE is trained to reconstruct the input images, ensuring the latent space captures meaningful scene information.
  2. Neural Radiance Field (NeRF) Decoder: After the PC-VAE is trained, its decoder is replaced with a NeRF. This NeRF is conditioned on the latent representations from the PC-VAE. This allows CARFF to render novel views of the scene, effectively "seeing" into occluded regions and providing a rich 3D representation.
  3. Mixture Density Network (MDN) for Forecasting: To predict future scene evolution, an MDN is used. It takes the latent representation of the current scene and predicts a distribution over possible future latent states. This probabilistic prediction is crucial for handling uncertainty. The MDN outputs a mixture of Gaussians, allowing it to represent multiple plausible future scenarios.

The overall training is a two-stage process: first, the PC-VAE is trained for reconstruction, and then the NeRF decoder is trained using the learned latents. The forecasting module (MDN) is trained on top of the frozen PC-VAE encoder.

Datasets

  • Blender Synthetic Dataset: Used for initial principle experiments to test the probabilistic modeling capabilities in isolation from complex vision encoders. It's visually simple but requires the full predictive model.
  • CARLA Driving Simulator Datasets:
    • Single-Scene Approaching Intersection: Simulates an ego vehicle at a T-intersection with an actor vehicle. Used for full observation prediction.
  • Multi-Scene Approaching Intersection: Extends the above with probabilistic actor existence and partial occlusions.
  • Multi-Scene Multi-actor Two Lane Merge: Introduces more dynamics uncertainty with multiple actors and partial occlusions.
  • Hand-manipulation Dataset: An additional dataset generated for hand-manipulation tasks involving reaching and grasping objects.

Key Insights and Ideas

  • Probabilistic 3D Scene Forecasting is Crucial for Planning: The paper strongly emphasizes that for safe autonomous driving, especially in occluded scenarios, predicting a distribution of possible future states is more important than a single deterministic prediction. This allows for contingency planning.
  • Pose-Conditioning for 3D Awareness: By conditioning the VAE encoder on camera pose, the model learns latent representations that are more invariant to viewpoint and better capture the underlying 3D structure of the scene. This is a key step towards making the latents truly 3D-aware.
  • Two-Stage Training Stabilizes Complex Models: Integrating VAEs and NeRFs can be challenging due to optimization complexities. The proposed two-stage training (VAE for latents, then NeRF for rendering) helps stabilize the process and achieve better results.
  • MDN for Multi-Modal Future Predictions: Using a Mixture Density Network is essential for capturing the multi-modal nature of future possibilities in dynamic environments, especially when dealing with occlusions and uncertain actor behaviors.

Key Results

  • High-Fidelity Reconstruction: The PC-VAE component demonstrates strong reconstruction capabilities, achieving high PSNR values (around 29) for predicted scenes in fully observable scenarios.
  • Effective Latent Space Separation: t-SNE visualizations and SVM classification show that the learned latent representations are separable by timestamps and scenes, indicating good view-invariance and ability to capture environmental states.
  • Accurate Forecasting: CARFF achieves high accuracy and recall in predicting scene states, even under partial observability and dynamic uncertainty, as shown in the Multi-Scene dataset experiments.
  • Improved Planning: The CARFF-based controller demonstrates effective decision-making, balancing safety and efficiency by adapting to uncertainty and occlusions. For instance, it can correctly choose to pause when actor presence is uncertain.
  • Qualitative Comparisons: CARFF outperforms related methods like NeRF-VAE in handling complex scenarios and probabilistic predictions, though direct quantitative comparisons are difficult due to differing objectives.

Limitations and Future Work

  • Scene-Specific Reliance: Like many NeRF-based methods, CARFF currently relies on posed images of specific scenes (e.g., intersections). Its direct applicability to entirely unseen environments is limited.
  • Complex Dynamics Handling: While CARFF handles multi-agent dynamics, extremely complex scenarios with a very large number of actors still pose a challenge, potentially requiring per-scene optimization.
  • Future Work: The authors suggest that enhanced generalizability could be achieved with more diverse camera deployments. They also anticipate future work on stronger models for handling more complex dynamics. The paper also mentions extending their application to a hand-manipulation dataset.

Summary updated

Abstract

We propose CARFF, a method for predicting future 3D scenes given past observations. Our method maps 2D ego-centric images to a distribution over plausible 3D latent scene configurations and predicts the evolution of hypothesized scenes through time. Our latents condition a global Neural Radiance Field (NeRF) to represent a 3D scene model, enabling explainable predictions and straightforward downstream planning. This approach models the world as a POMDP and considers complex scenarios of uncertainty in environmental states and dynamics. Specifically, we employ a two-stage training of Pose-Conditional-VAE and NeRF to learn 3D representations, and auto-regressively predict latent scene representations utilizing a mixture density network. We demonstrate the utility of our method in scenarios using the CARLA driving simulator, where CARFF enables efficient trajectory and contingency planning in complex multi-agent autonomous driving scenarios involving occlusions.
BibTeX
@misc{yang2024carff,
  author = {Jiezhi Yang and Khushi Desai and Charles Packer and Harshil Bhatia and Nicholas Rhinehart and Rowan McAllister and Joseph Gonzalez},
  title = {{CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecasting}},
  year = {2024},
  eprint = {2401.18075},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.18075},
}
Figure