Ray Priors through Reprojection: Improving Neural Radiance Fields for Novel View Extrapolation

Jian Zhang, Yuanqing Zhang, Huan Fu, Xiaowei Zhou, Bowen Cai, Jinchi Huang, Rongfei Jia, Binqiang Zhao, Xing Tang

Neural Radiance Fields (NeRF) have revolutionized how we represent 3D scenes, but they have a well-known "Achilles' heel": they are excellent at interpolating between known viewpoints but often fall apart when asked to extrapolate—that is, to render the scene from a viewpoint significantly different from any seen during training.

In their paper, "Ray Priors through Reprojection," researchers from Alibaba Group and Zhejiang University introduce RapNeRF, a solution that leverages multi-view consistency to bridge the gap between interpolation and extrapolation.

TL;DR

RapNeRF improves NeRF’s ability to render novel views far from the training distribution by introducing two key techniques: Random Ray Casting (RRC) and a Ray Atlas (RA). By enforcing the idea that a 3D surface point should look consistent regardless of the viewing angle, RapNeRF significantly reduces artifacts in extrapolated views, outperforming standard NeRF by nearly 5 dB in PSNR on challenging real-world datasets.

What Problem Does This Paper Address?

Standard NeRF models learn a mapping from a 5D coordinate (location $x, y, z$ and viewing direction $\theta, \phi$) to color and density. While this works beautifully when the test camera is near the training cameras, it struggles with distribution shift.

When a test ray comes from a direction far outside the training set, the model’s "direction embedding" often produces unpredictable results, leading to "ripple" artifacts, non-smooth colors, or complete geometry breakdown. The authors distinguish this from "few-shot" learning; here, the training images might describe the object perfectly, but the viewpoints are clustered in one area (e.g., only the front of an object), making the back or side views "extrapolated" territory.

Figure 1: The extrapolation setting. Training views (red) are clustered, while test views (blue) are distant. NeRF produces artifacts in these regions, while RapNeRF remains clean.

Methods

RapNeRF introduces two primary mechanisms to enforce multi-view consistency during training:

1. Random Ray Casting (RRC)

The core idea of RRC is to supervise the model on viewpoints it has never seen. During training, for a ray that hits a surface point $v$, the algorithm randomly generates a "virtual" ray from a new origin $o'$ that also passes through $v$. Because of the multi-view consistency property, the model is told that this virtual ray should have the same color as the original training ray. This acts as an online data augmentation strategy in ray-space, teaching the network how the surface looks from unseen angles.

Figure 2: Random Ray Casting (right) generates virtual rays within a cone to provide pseudo-labels for unseen viewpoints.

2. Ray Atlas (RA)

To further stabilize rendering, the authors propose a Ray Atlas. They extract a rough 3D mesh (R3DM) from a pre-trained NeRF and, for every vertex, compute a "global ray prior"—essentially an average of all training ray directions that hit that point. During fine-tuning and inference, the model uses this averaged direction for its color prediction instead of the raw, potentially noisy viewing direction. This helps smooth out the surface appearance and prevents the "rippling" effect common in extrapolation.

Figure 3: The Ray Atlas textures a rough mesh with averaged training ray directions to provide a stable global prior.

Datasets

The authors evaluated RapNeRF on two main benchmarks:

  • Synthetic-NeRF*: A re-split of the classic Blender dataset where cameras are sorted by their Z-axis position. The first 100 views are used for training, and the remaining 200 (often from very different heights/angles) are used for testing.
  • MobileObject: A new dataset introduced in this paper consisting of 8 real-world objects captured via mobile phone. The test views are specifically chosen to have a large "pose distance" from the training set to ensure a true extrapolation challenge.

Key Insights and Ideas

  • Consistency over Specularity: The fundamental "aha" moment is that for extrapolation, multi-view consistency is a more reliable prior than the complex view-dependent effects (like specular highlights) that NeRF usually tries to capture. By forcing consistency, you lose some "shininess," but you gain massive stability.
  • RRC as Regularization: RRC effectively regularizes the color prediction MLP, preventing it from overfitting to the specific viewing directions present in the training set.
  • Geometry First: The method uses a two-stage approach—first learning the geometry with a standard NeRF, then fine-tuning the appearance with RRC and RA. This ensures the "virtual rays" in RRC actually hit the correct surface points.

Key Results

RapNeRF shows a dramatic improvement over both the original NeRF and recent variants like NSVF and IBRNet in extrapolation scenarios.

  • Quantitative: On the MobileObject dataset, RapNeRF achieved a PSNR of 28.90, compared to 24.05 for standard NeRF.
  • Qualitative: As seen in Figure 4, RapNeRF recovers fine details and smooth surfaces where other methods produce significant noise or "floaters."
  • Robustness: The authors demonstrated that as the "pose distance" (the difference between training and test angles) increases, RapNeRF’s performance remains relatively stable, while other methods' quality drops sharply.
Figure 4: Qualitative comparison showing RapNeRF's superior detail recovery in extrapolated views compared to SOTA methods.

Limitations and Future Work

The primary limitation is the trade-off between extrapolation and view-dependent effects. Because RapNeRF enforces multi-view consistency, it tends to "bake in" lighting, removing strong specularities.

To address this, the authors proposed a variant called RapNeRF†, which uses a deferred rendering architecture (similar to SNeRG) to add a small MLP that predicts a specular component. While this helps recover some shininess, it slightly reduces the overall PSNR compared to the "pure" RapNeRF.

Future work includes applying these ray priors to sparse-view reconstruction (few-shot NeRF), where the lack of data makes consistency priors even more critical.

Summary updated

Abstract

Neural Radiance Fields (NeRF) have emerged as a potent paradigm for representing scenes and synthesizing photo-realistic images. A main limitation of conventional NeRFs is that they often fail to produce high-quality renderings under novel viewpoints that are significantly different from the training viewpoints. In this paper, instead of exploiting few-shot image synthesis, we study the novel view extrapolation setting that (1) the training images can well describe an object, and (2) there is a notable discrepancy between the training and test viewpoints' distributions. We present RapNeRF (RAy Priors) as a solution. Our insight is that the inherent appearances of a 3D surface's arbitrary visible projections should be consistent. We thus propose a random ray casting policy that allows training unseen views using seen views. Furthermore, we show that a ray atlas pre-computed from the observed rays' viewing directions could further enhance the rendering quality for extrapolated views. A main limitation is that RapNeRF would remove the strong view-dependent effects because it leverages the multi-view consistency property.
BibTeX
@misc{zhang2022ray,
  author = {Jian Zhang and Yuanqing Zhang and Huan Fu and Xiaowei Zhou and Bowen Cai and Jinchi Huang and Rongfei Jia and Binqiang Zhao and Xing Tang},
  title = {{Ray Priors through Reprojection: Improving Neural Radiance Fields for Novel View Extrapolation}},
  year = {2022},
  eprint = {2205.05922},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.05922},
}
Figure