GUMBEL-NERF: Representing Unseen Objects as Part-Compositional Neural Radiance Fields

Yusuke Sekikawa, Chingwei Hsu, Satoshi Ikehata, Rei Kawakami, Ikuro Sato

Introduction

Reconstructing 3D objects from a single 2D image—often referred to as one-shot novel view synthesis—remains a cornerstone challenge in computer vision. While models like CodeNeRF have made strides by learning category-level latent codes, they often struggle to capture the intricate geometric diversity of complex objects. To scale model capacity, researchers have explored Mixture-of-Experts (MoE) architectures, such as Switch-NeRF, which partition scenes among specialized sub-networks.

However, traditional MoE-NeRFs rely on "foresight" gating. In this setup, a separate gating network predicts which expert to use for a specific 3D coordinate before evaluating the experts themselves. For novel, unseen objects, this prediction is frequently inaccurate, leading to "seams" or visible discontinuities where different experts fail to align at their boundaries. Gumbel-NeRF addresses this by introducing a "hindsight" selection mechanism that ensures geometric continuity and enables high-quality reconstruction from minimal input.

Methodology: Hindsight Expert Selection

The core innovation of Gumbel-NeRF is the shift from predicting expertise to observing it through a density-based selection process.

1. Density as a Proxy for Expertise

Instead of relying on a preliminary gate, Gumbel-NeRF passes a 3D point through all $N$ experts simultaneously. The model then selects the expert that produces the highest volume density ($\sigma$) at that specific point. Because NeRFs represent geometry through density, the expert with the highest density is effectively the one that has the most "knowledge" or "claim" over that part of the 3D space. By using a maximum-pooling approach over densities, the resulting field remains continuous, eliminating the structural breaks seen in foresight-based models.

2. Part-Specific Latent Codes

To encourage specialization, each expert is associated with its own latent code. This architecture allows the model to learn a part-compositional representation. Without explicit labels, different experts naturally begin to specialize in specific components—for example, one expert might consistently represent wheels across various car models, while another focuses on the chassis or windows.

3. The Rival-to-Expert Strategy

A common failure mode in MoE models is "router collapse," where a single expert becomes a generalist and the others remain untrained. Gumbel-NeRF prevents this using the Gumbel-Max trick and a "Rival-to-Expert" training schedule:

  • Rival Stage: Early in training, a high temperature ($\tau$) parameter is used, making the selection process nearly random. This forces all experts to attempt to represent the general shape of the object.
  • Expert Stage: As training progresses, $ au$ is annealed using a cosine schedule. The selection becomes more deterministic, allowing experts to specialize in the specific regions or parts they represent most accurately.

Experimental Results

The authors evaluated Gumbel-NeRF on the ShapeNet-SRN cars dataset, a standard benchmark for category-level 3D reconstruction. The model was trained on 2,458 car instances and tested on 704 unseen instances.

Quantitative Performance

Gumbel-NeRF demonstrated a significant leap in performance over both global-code baselines and standard MoE architectures:

  • Peak Signal-to-Noise Ratio (PSNR): Gumbel-NeRF achieved 21.51, substantially higher than CodeNeRF (19.66) and the Coded Switch-NeRF baseline (19.50).
  • Perceptual Quality (LPIPS): The model reduced the LPIPS score to 0.119, compared to approximately 0.150 for the baselines, indicating that the reconstructed images are much closer to human perception of the ground truth.

Qualitative Advantages

Beyond the numbers, Gumbel-NeRF produces visually superior results. While previous MoE models often suffered from "ghosting" or fragmented geometry, Gumbel-NeRF generates sharp, coherent 3D structures. The emergent decomposition is particularly noteworthy; the model successfully segments objects into logical parts without any manual supervision, providing a more interpretable and modular 3D representation.

Discussion and Limitations

While Gumbel-NeRF sets a new standard for one-shot reconstruction, it introduces a trade-off in computational efficiency. Because the hindsight mechanism requires a forward pass through all experts to determine the winner, the inference cost scales linearly with the number of experts. This makes it more "dense" than "sparse" MoE models that only activate a single expert.

Future work may focus on optimizing this computational overhead or scaling the part-compositional approach to more complex, multi-category scenes. The ability to decompose objects into parts also opens doors for advanced scene editing, such as swapping components between different reconstructed objects.

Conclusion

Gumbel-NeRF demonstrates that by simply changing when an expert is selected, we can solve fundamental continuity issues in MoE-based neural representations. By leveraging density as a proxy for expertise, the model achieves state-of-the-art results in one-shot 3D reconstruction and paves the way for more robust, part-aware generative models.

Summary updated

Abstract

We propose Gumbel-NeRF, a mixture-of-expert (MoE) neural radiance fields (NeRF) model with a hindsight expert selection mechanism for synthesizing novel views of unseen objects. Previous studies have shown that the MoE structure provides high-quality representations of a given large-scale scene consisting of many objects. However, we observe that such a MoE NeRF model often produces low-quality representations in the vicinity of experts' boundaries when applied to the task of novel view synthesis of an unseen object from one/few-shot input. We find that this deterioration is primarily caused by the foresight expert selection mechanism, which may leave an unnatural discontinuity in the object shape near the experts' boundaries. Gumbel-NeRF adopts a hindsight expert selection mechanism, which guarantees continuity in the density field even near the experts' boundaries. Experiments using the SRN cars dataset demonstrate the superiority of Gumbel-NeRF over the baselines in terms of various image quality metrics.
BibTeX
@inproceedings{sekikawa2024gumbel,
  author = {Yusuke Sekikawa and Chingwei Hsu and Satoshi Ikehata and Rei Kawakami and Ikuro Sato},
  title = {{GUMBEL-NERF: Representing Unseen Objects as Part-Compositional Neural Radiance Fields}},
  booktitle = {ICIP},
  year = {2024},
  doi = {10.1109/icip51287.2024.10648036},
}
Figure