Gumbel-NeRF: Representing Unseen Objects as Part-Compositional Neural Radiance Fields

Yusuke Sekikawa, Chingwei Hsu, Satoshi Ikehata, Rei Kawakami, Ikuro Sato

Introduction

Reconstructing 3D objects from a single 2D image—often referred to as one-shot novel view synthesis—remains a cornerstone challenge in computer vision. While models like CodeNeRF have made strides by learning category-level latent codes, they often struggle to capture the intricate geometric diversity of complex objects. To scale model capacity, researchers have explored Mixture-of-Experts (MoE) architectures, such as Switch-NeRF, which partition scenes among specialized sub-networks.

However, traditional MoE-NeRFs rely on "foresight" gating. In this setup, a separate gating network predicts which expert to use for a specific 3D coordinate before evaluating the experts themselves. For novel, unseen objects, this prediction is frequently inaccurate, leading to "seams" or visible discontinuities where different experts fail to align at their boundaries. Gumbel-NeRF addresses this by introducing a "hindsight" selection mechanism that ensures geometric continuity and enables high-quality reconstruction from minimal input.

Methodology: Hindsight Expert Selection

The core innovation of Gumbel-NeRF is the shift from predicting expertise to observing it through a density-based selection process.

1. Density as a Proxy for Expertise

Instead of relying on a preliminary gate, Gumbel-NeRF passes a 3D point through all $N$ experts simultaneously. The model then selects the expert that produces the highest volume density ($\sigma$) at that specific point. Because NeRFs represent geometry through density, the expert with the highest density is effectively the one that has the most "knowledge" or "claim" over that part of the 3D space. By using a maximum-pooling approach over densities, the resulting field remains continuous, eliminating the structural breaks seen in foresight-based models.

2. Part-Specific Latent Codes

To encourage specialization, each expert is associated with its own latent code. This architecture allows the model to learn a part-compositional representation. Without explicit labels, different experts naturally begin to specialize in specific components—for example, one expert might consistently represent wheels across various car models, while another focuses on the chassis or windows.

3. The Rival-to-Expert Strategy

A common failure mode in MoE models is "router collapse," where a single expert becomes a generalist and the others remain untrained. Gumbel-NeRF prevents this using the Gumbel-Max trick and a "Rival-to-Expert" training schedule:

  • Rival Stage: Early in training, a high temperature ($\tau$) parameter is used, making the selection process nearly random. This forces all experts to attempt to represent the general shape of the object.
  • Expert Stage: As training progresses, $ au$ is annealed using a cosine schedule. The selection becomes more deterministic, allowing experts to specialize in the specific regions or parts they represent most accurately.

Experimental Results

The authors evaluated Gumbel-NeRF on the ShapeNet-SRN cars dataset, a standard benchmark for category-level 3D reconstruction. The model was trained on 2,458 car instances and tested on 704 unseen instances.

Quantitative Performance

Gumbel-NeRF demonstrated a significant leap in performance over both global-code baselines and standard MoE architectures:

  • Peak Signal-to-Noise Ratio (PSNR): Gumbel-NeRF achieved 21.51, substantially higher than CodeNeRF (19.66) and the Coded Switch-NeRF baseline (19.50).
  • Perceptual Quality (LPIPS): The model reduced the LPIPS score to 0.119, compared to approximately 0.150 for the baselines, indicating that the reconstructed images are much closer to human perception of the ground truth.

Qualitative Advantages

Beyond the numbers, Gumbel-NeRF produces visually superior results. While previous MoE models often suffered from "ghosting" or fragmented geometry, Gumbel-NeRF generates sharp, coherent 3D structures. The emergent decomposition is particularly noteworthy; the model successfully segments objects into logical parts without any manual supervision, providing a more interpretable and modular 3D representation.

Discussion and Limitations

While Gumbel-NeRF sets a new standard for one-shot reconstruction, it introduces a trade-off in computational efficiency. Because the hindsight mechanism requires a forward pass through all experts to determine the winner, the inference cost scales linearly with the number of experts. This makes it more "dense" than "sparse" MoE models that only activate a single expert.

Future work may focus on optimizing this computational overhead or scaling the part-compositional approach to more complex, multi-category scenes. The ability to decompose objects into parts also opens doors for advanced scene editing, such as swapping components between different reconstructed objects.

Conclusion

Gumbel-NeRF demonstrates that by simply changing when an expert is selected, we can solve fundamental continuity issues in MoE-based neural representations. By leveraging density as a proxy for expertise, the model achieves state-of-the-art results in one-shot 3D reconstruction and paves the way for more robust, part-aware generative models.

Summary updated

BibTeX
@inproceedings{sekikawa2024gumbel,
  author = {Yusuke Sekikawa and Chingwei Hsu and Satoshi Ikehata and Rei Kawakami and Ikuro Sato},
  title = {{Gumbel-NeRF: Representing Unseen Objects as Part-Compositional Neural Radiance Fields}},
  booktitle = {ICIP},
  year = {2024},
  doi = {10.1109/icip51287.2024.10648036},
}
Figure