Abstract
Realistic simulation is key to enabling safe and scalable development of self-driving vehicles. A core component is simulating the sensors so that the entire autonomy system can be tested in simulation. Sensor simulation involves modeling traffic participants, such as vehicles, with high quality appearance and articulated geometry, and rendering them in real time. The self-driving industry has typically employed artists to build these assets. However, this is expensive, slow, and may not reflect reality. Instead, reconstructing assets automatically from sensor data collected in the wild would provide a better path to generating a diverse and large set with good real-world coverage. Nevertheless, current reconstruction approaches struggle on in-the-wild sensor data, due to its sparsity and noise. To tackle these issues, we present CADSim, which combines part-aware object-class priors via a small set of CAD models with differentiable rendering to automatically reconstruct vehicle geometry, including articulated wheels, with high-quality appearance. Our experiments show our method recovers more accurate shapes from sparse data compared to existing approaches. Importantly, it also trains and renders efficiently. We demonstrate our reconstructed vehicles in several applications, including accurate testing of autonomy perception systems.
TL;DR
CADSim is a robust framework that automatically reconstructs high-fidelity, articulated 3D vehicle models from sparse and noisy "in-the-wild" sensor data (camera and LiDAR). By leveraging CAD model priors and differentiable rendering, it creates simulation-ready assets that are not only photorealistic but also controllable, allowing for realistic steering and wheel rotation in virtual environments.
Introduction
The development of self-driving vehicles (SDVs) relies heavily on high-fidelity simulation environments to test autonomy stacks safely and at scale. A critical bottleneck in this process is the creation of 3D assets—such as cars, trucks, and buses—that populate these virtual worlds. Traditionally, these assets are handcrafted by human artists, a process that is not only prohibitively expensive and slow but also fails to capture the vast diversity of vehicles encountered in the real world.
While automated reconstruction from sensor data (camera and LiDAR) offers a promising alternative, "in-the-wild" data presents significant challenges. Such data is typically sparse, noisy, and captured from limited viewpoints. Existing methods, including Neural Radiance Fields (NeRFs), often produce visually appealing results but fail to generate the precise, articulated geometry required for physical simulation. CADSim addresses these challenges by bridging the gap between raw sensor logs and controllable digital twins.
Methodology
CADSim integrates class-specific CAD priors with differentiable rendering to reconstruct high-quality vehicle models from sparse inputs. The framework is built upon four key pillars:
1. CAD-based Latent Space
Rather than optimizing a generic shape, CADSim utilizes a library of CAD models to learn a low-dimensional "shape space" via Principal Component Analysis (PCA). This prior knowledge ensures that the reconstructed mesh always maintains the structural integrity of a valid vehicle, even when data for certain parts (like the undercarriage) is missing.
2. Part-Aware Articulation
The model explicitly parameterizes the vehicle into a body and four wheels. This allows the system to recover not just the overall shape, but also specific attributes like wheel radius, position, and steering angle, which are essential for realistic animation.
3. Energy-Based Optimization
The reconstruction is treated as an optimization problem, minimizing a multi-term energy function. This includes visual terms (RGB color and segmentation masks), geometric terms (aligning the mesh with LiDAR point clouds using Chamfer Distance), and regularization terms to ensure physical plausibility and smoothness.
4. Differentiable Rendering
By employing differentiable rendering tools like Nvdiffrast, CADSim can backpropagate errors from 2D image pixels directly to 3D mesh vertices and material properties. This enables the system to refine the vehicle's appearance and geometry simultaneously.
Experimental Evaluation
The authors evaluated CADSim on two primary datasets: the Multi-View Marketplace Cars (MVMC) and PandaVehicle (a subset of PandaSet). The latter represents a particularly challenging real-world scenario with sparse camera views and aggregated LiDAR data.
Key Results
- Novel View Synthesis: On PandaSet, CADSim achieved a PSNR of 21.72, significantly outperforming NeRF++ (20.86) and NeRS (18.49). It also produced fewer visual artifacts, as evidenced by a superior LPIPS score of 0.220.
- Geometric Accuracy: CADSim's geometry is far more accurate for LiDAR simulation, achieving a Chamfer distance of 0.245, compared to 0.424 for NeuS and 0.439 for NVDiffRec.
- Perception System Testing: When used to test object detectors, CADSim-generated data showed a much higher correlation with real-world performance than competing methods, proving its utility for autonomy testing.
- Efficiency: The reconstruction process takes less than 10 minutes, and the resulting assets support real-time rendering at approximately 50 FPS on standard hardware.
Conclusion and Future Work
CADSim demonstrates that combining object-class priors with differentiable rendering is a powerful approach for scalable 3D asset generation. While the current system relies on a CAD library for template initialization, it successfully generalizes to a wide variety of vehicle shapes. Future research will focus on extending this framework to other classes, such as pedestrians and cyclists, and automating the rigging of additional parts like doors and trunks to further enhance simulation realism.
Summary updated