CMC: Few-shot Novel View Synthesis via Cross-view Multiplane Consistency

Hanxin Zhu, Zhibo Chen

Abstract

Novel view synthesis from sparse input images remains a significant challenge in computer vision. While Neural Radiance Fields (NeRF) have achieved remarkable results with dense inputs, they frequently fail in "few-shot" scenarios by overfitting to the training views and producing degenerate, flat geometry. This paper introduces CMC (Cross-view Multiplane Consistency), a framework that addresses these limitations by shifting from continuous volume representations to discrete, per-view Multiplane Images (MPIs). By enforcing appearance and depth consistency across multiple MPIs, CMC regularizes the 3D structure without requiring external supervision, achieving state-of-the-art performance on challenging benchmarks.

Introduction

The ability to reconstruct a 3D scene from a limited set of 2D images is a cornerstone of immersive technology. However, standard NeRF models are notoriously "data-hungry." When provided with as few as three views, the optimization process often collapses. Instead of learning the true 3D structure of the scene, the model simply memorizes the pixel colors of the input rays—a phenomenon known as overfitting. This results in "floater" artifacts, blurry novel views, and depth maps that lack physical meaning.

CMC proposes a geometric solution to this sampling problem. By discretizing the scene into a set of fixed planes and enforcing consistency between different viewpoints, the model is forced to find a consensus on the scene's geometry, leading to significantly more robust synthesis.

The Challenge of Sparse Views

The authors identify a fundamental geometric reason for NeRF's failure in few-shot settings: the lack of ray intersections. In a continuous volume, rays cast from sparse viewpoints rarely intersect at the exact same sampling points. Without these intersections, the neural network receives no signal to ensure that different views agree on the same spatial point. Consequently, the model defaults to the simplest solution—placing all color information on a flat surface near the camera—which satisfies the training views but fails to generalize to novel perspectives.

The CMC Framework

CMC departs from the single global volume approach and instead utilizes a multi-MPI representation:

  1. Per-view MPIs: The system constructs an individual Multiplane Image for every input view. Each MPI consists of a series of fronto-parallel planes at fixed depths. These planes are parameterized by a small Multi-Layer Perceptron (MLP) that predicts color and density.
  2. Deterministic Sampling: By forcing the scene representation onto fixed, discrete planes, the "sampling points" become deterministic. This significantly increases the likelihood that rays from different views will hit the same spatial locations, facilitating better interaction and geometric agreement between views.
  3. Weighted Rendering: To render a novel view, CMC performs a weighted average of the outputs from all available MPIs. The weight of each MPI is determined by the proximity of its reference camera to the target viewpoint, ensuring that the most relevant geometric information is prioritized.

Cross-view Consistency Losses

The core innovation of CMC is the introduction of two "internal" consistency losses that regularize the model for unseen views without relying on pre-trained depth estimators or external models:

  • Appearance Consistency ($L_{ac}$): This loss ensures that a novel view rendered from MPI A is identical to the same view rendered from MPI B. If the two MPIs disagree on the color of a point in space, the model is penalized.
  • Depth Consistency ($L_{dc}$): Similarly, the depth map of a novel view must remain consistent regardless of which input MPI was used to generate it. This forces the model to converge on a single, physically plausible geometry for the entire scene.

Experimental Results

The researchers evaluated CMC against several baselines, including vanilla NeRF and specialized few-shot models like DietNeRF and RegNeRF, using the LLFF and Shiny datasets in an extremely sparse 3-view setting.

Quantitative Performance

CMC achieved state-of-the-art results across both benchmarks:

  • Shiny Dataset: CMC reached a PSNR of 15.98, a substantial improvement over vanilla NeRF (14.37) and DietNeRF (13.12). This dataset is particularly difficult due to complex reflections and refractions.
  • LLFF Dataset: CMC achieved a PSNR of 19.45, outperforming the previous leading method, RegNeRF (19.08).

Qualitative Improvements

The most notable difference is visible in the estimated depth maps. While competing methods often produce noisy or "flat" depth estimations, CMC generates smooth, continuous geometry. It successfully reconstructs thin structures and handles occlusions that typically cause artifacts in other few-shot frameworks.

Conclusion and Future Work

CMC demonstrates that discrete layered representations (MPIs) offer unique advantages for geometric regularization that continuous volumes struggle to provide in sparse-data environments. By prioritizing internal consistency, the model achieves high-quality synthesis without external priors.

Currently, CMC is limited to forward-facing scenes due to the nature of MPIs. Future research directions include extending the framework to 360-degree scenes using Multi-sphere Images and optimizing the computational efficiency of the per-view MLP architecture.

Summary updated

Abstract

Neural Radiance Field (NeRF) has shown impressive results in novel view synthesis, particularly in Virtual Reality (VR) and Augmented Reality (AR), thanks to its ability to represent scenes continuously. However, when just a few input view images are available, NeRF tends to overfit the given views and thus make the estimated depths of pixels share almost the same value. Unlike previous methods that conduct regularization by introducing complex priors or additional supervisions, we propose a simple yet effective method that explicitly builds depth-aware consistency across input views to tackle this challenge. Our key insight is that by forcing the same spatial points to be sampled repeatedly in different input views, we are able to strengthen the interactions between views and therefore alleviate the overfitting problem. To achieve this, we build the neural networks on layered representations (i.e., multiplane images), and the sampling point can thus be resampled on multiple discrete planes. Furthermore, to regularize the unseen target views, we constrain the rendered colors and depths from different input views to be the same. Although simple, extensive experiments demonstrate that our proposed method can achieve better synthesis quality over state-of-the-art methods.
BibTeX
@inproceedings{zhu2024cmc,
  author = {Hanxin Zhu and Zhibo Chen},
  title = {{CMC: Few-shot Novel View Synthesis via Cross-view Multiplane Consistency}},
  booktitle = {VR},
  year = {2024},
  url = {https://openreview.net/forum?id=3yFoONqXWj},
}
Figure