Variable Radiance Field for Real-World Category-Specific Reconstruction from Single Image

Kun Wang, Zhiqiang Yan, Zhenyu Zhang, Xiang Li, Jun Li, Jian YangSSRN 2025 · DOI

TL;DR

This paper introduces Variable Radiance Field (VRF), a novel framework for reconstructing 3D objects from a single image without requiring camera intrinsics. VRF achieves this by using global latent representations for geometry and appearance, a category-specific shape template for alignment, and a hyper-network for efficient rendering, leading to state-of-the-art results in reconstruction quality and speed.

What Problem Does This Paper Address?

Reconstructing 3D objects from a single image is a challenging but highly desirable task for applications like AR/VR and animation. Existing methods, often based on Neural Radiance Fields (NeRF), typically rely on projecting 3D points onto a reference image to retrieve local features. This approach has several drawbacks: it's computationally expensive, requires known camera intrinsics (which are often unavailable for real-world images), and is highly susceptible to occlusions, leading to artifacts when the target view significantly differs from the input. Furthermore, real-world datasets often have inconsistent camera poses and scales across different object instances, further complicating reconstruction.

Methods

The VRF framework tackles these challenges with three main components:

  • Object Encoding Module (OEM): This module replaces traditional projection-based feature retrieval with a global encoding scheme. It uses a convolutional feature extractor to generate multi-level features from the input image and its mask. These features are then processed to create two latent vectors: one for geometry ($z_d$) and one for appearance ($z_c$). This global approach eliminates the need for camera intrinsics and is more robust to occlusions. To improve feature extraction, a contrastive learning-based pre-training strategy is employed, encouraging similar representations for different views of the same object and dissimilar ones for different objects.
  • Dynamic Ray Sampling Module (DRSM): To address the coordinate inconsistency in real-world datasets, VRF introduces a category-specific shape template. Each training object instance is then aligned to this canonical template using a learned 3D transformation. This alignment simplifies geometry prediction by focusing on deformations from the template to individual instances. During training, the template is fine-tuned to represent the average shape of the category.
  • Instance Creation Module (ICM): This module generates a compact, instance-specific NeRF network for efficient rendering. It uses a hyper-network-based approach where two hyper-networks generate the weights for a deformation submodule and a color submodule. The deformation submodule predicts instance-specific shape deformations relative to the template, and the color submodule predicts color based on view direction and feature maps. This design allows for fast rendering without needing large, category-specific models at inference time.

Datasets

The experiments were conducted on the CO3D dataset, which contains images of various object categories. The authors followed the official data split for training and testing.

Key Insights and Ideas

  • Global features over local features: Moving from local, projection-based feature retrieval to global latent representations significantly improves efficiency and robustness, especially in the presence of occlusions and unknown camera intrinsics.
  • Canonical space for consistency: Defining a category-specific shape template and aligning instances to it provides a crucial anchor for handling the inherent coordinate inconsistencies in real-world datasets. This simplifies the geometry learning problem.
  • Hyper-networks for efficient rendering: Using hyper-networks to generate instance-specific NeRF weights allows for much faster rendering compared to traditional NeRF or category-specific models, making the approach more practical.
  • Contrastive pre-training is key: The contrastive learning pre-training strategy for the feature extractor is vital for learning discriminative features that can effectively disentangle geometry and appearance, leading to better reconstruction quality.

Key Results

  • State-of-the-art performance: VRF outperforms existing methods like pixelNeRF, Henzler et al., GRF, NerFormer, and VisionNeRF in reconstruction quality across multiple categories (banana, cellphone, hydrant, mouse, cup) based on PSNR, SSIM, and LPIPS metrics. For instance, on the Hydrant category, it achieves a 7.58% improvement in PSNR over GRF.
  • Faster rendering speed: VRF achieves significantly faster rendering speeds, with an average of 0.75 seconds per frame on a TITAN RTX GPU, outperforming most competitors. It is 21.9% faster than Henzler et al. while achieving a 42.7% higher average PSNR.
  • Robustness to viewpoint changes: The method maintains consistent performance even with large viewpoint differences between the input and target images, as demonstrated by experiments in Figure 6.
  • Qualitative improvements: Visual comparisons show that VRF produces sharper, more 3D-consistent reconstructions with clearer foreground-background boundaries, especially for challenging objects with non-Lambertian surfaces.

Limitations and Future Work

The authors acknowledge that the current method is based on the original NeRF, which can still be constrained by slow rendering speeds. While their hyper-network approach offers acceleration, they plan to integrate even faster techniques like instant-ngp and 3D Gaussian splatting to further optimize rendering speed for real-world applications. Additionally, while the current focus is on object reconstruction, they believe the method is well-suited for human reconstruction and intend to explore this in future research.

Summary updated

BibTeX
@misc{wang2025variable,
  author = {Kun Wang and Zhiqiang Yan and Zhenyu Zhang and Xiang Li and Jun Li and Jian Yang},
  title = {{Variable Radiance Field for Real-World Category-Specific Reconstruction from Single Image}},
  year = {2025},
  doi = {10.2139/ssrn.5162593},
  howpublished = {SSRN},
}
Figure