GenLayNeRF: Generalizable Layered Representations with 3D Model Alignment for Human View Synthesis

Youssef Abdelkareem, Shady Shehata, Fakhri Karray

Synthesizing new viewpoints of people in motion—a task known as Novel View Synthesis (NVS)—is a cornerstone of the future of telepresence and VR. While Neural Radiance Fields (NeRF) have revolutionized this field, they usually struggle when multiple people interact, occlude one another, or when the system needs to work on a new person it hasn't seen before.

GenLayNeRF, a new framework from researchers at the University of Waterloo and MBZUAI, tackles these challenges head-on by combining layered scene representations with a clever iterative alignment strategy.

TL;DR

GenLayNeRF is a generalizable model for synthesizing novel views of multiple interacting humans from very sparse (3-4) camera views. By using an end-to-end alignment module to fix errors in 3D body models and a layered architecture to handle occlusions, it achieves high-quality results on new subjects without requiring hours of per-scene optimization.

What Problem Does This Paper Address?

Current human-centric NVS methods face three major hurdles:

  1. The Multi-Person Mess: When people interact closely (e.g., boxing or dancing), they occlude each other. Standard NeRFs often get "confused" about which pixels belong to which person.
  2. The Efficiency Gap: Many high-quality methods require "per-scene optimization," meaning you have to train the model for hours on a specific video before you can render a new view. This isn't practical for real-time applications.
  3. The Alignment Trap: Most methods rely on a 3D "skeleton" or mesh (like the SMPL model) to guide the NeRF. However, in sparse-view setups, these meshes are often slightly misaligned with the actual pixels. Even a tiny offset leads to blurry renders and "hallucinations" in the final image.

Methods

GenLayNeRF introduces a pipeline that is both layered and generalizable:

  • Layered Scene Representation: Instead of treating the whole room as one volume, the model divides the scene into layers—one for each human. Each layer is anchored by a 3D SMPL mesh. This allows the model to handle complex occlusions by rendering layers independently and then compositing them.
  • Human-Image Alignment Module: This is the paper's core innovation. Instead of taking a "pre-fitted" 3D mesh as gospel, the model uses an iterative feedback loop. It projects the mesh onto the image, extracts features, and then predicts a correction ($\Delta\Theta$) to the mesh parameters. This ensures the 3D model perfectly matches the 2D pixels before rendering begins.
  • Attention-Aware Fusion: Rather than simply averaging data from different camera angles, GenLayNeRF uses Self-Attention and Cross-Attention. This allows the network to "decide" which camera has the best view of a specific body part (e.g., ignoring a camera where an arm is hidden behind a torso).
  • RGB Fusion: To keep textures sharp, the model injects low-level RGB information directly into the feature fusion process, helping retain high-frequency details like clothing patterns.

Datasets

The authors noted a lack of diverse multi-human NVS benchmarks, so they introduced/adapted two:

  • DeepMultiSyn: Adapted from the DeepMultiCap dataset, featuring 2-3 humans in challenging scenarios like boxing and dancing.
  • ZJU-MultiHuman: A new dataset with 4 human subjects captured from 8 synchronized cameras.
  • ZJU-MoCap: They also used a subset of this single-human dataset to ensure the model maintains high performance on simpler scenes.

Key Insights and Ideas

  • Alignment is the Bottleneck: The researchers found that synthesis quality is highly sensitive to mesh accuracy. By making the alignment part of the end-to-end training, they significantly reduced the "ghosting" artifacts common in other models.
  • Layering Simplifies Geometry: By treating each person as a separate radiance field, the model doesn't have to learn the global geometry of the room; it only needs to learn how a human looks relative to their own body mesh.
  • Generalization via Features: Because the model learns to map image features to the 3D mesh rather than memorizing a specific scene, it can be applied to a totally new person in a new pose (Zero-Shot) and still produce a clean render.

Key Results

  • Superior Generalization: On the DeepMultiSyn dataset for "Unseen Models/Unseen Poses," GenLayNeRF achieved a PSNR of 21.03, significantly outperforming other generalizable baselines like NHP (19.51) and IBRNet (18.80).
  • Efficiency: While per-scene methods (like L-NeRF) take roughly 144 hours to train on a specific sequence, GenLayNeRF performs at a comparable level (and sometimes better) with zero test-time optimization.
  • Qualitative Sharpness: Visual results show that GenLayNeRF maintains much better limb definition and facial features compared to baselines, which often "melt" the subjects together during occlusions.

Limitations and Future Work

  • Clothing Constraints: The model relies on the SMPL mesh, which represents a "naked" human body. Consequently, it struggles with loose-fitting clothing like skirts or capes, which don't follow the body's topology closely.
  • Backgrounds: The current work focuses heavily on the human layers; integrating a similarly robust generalizable background model would be the next step for full scene synthesis.
  • Complex Models: The authors suggest moving toward more complex body models (like SMPL-X) to capture finger movements and facial expressions more accurately.

Summary updated

Abstract

Novel view synthesis (NVS) of multi-human scenes imposes challenges due to the complex inter-human occlusions. Layered representations handle the complexities by dividing the scene into multi-layered radiance fields, however, they are mainly constrained to per-scene optimization making them inefficient. Generalizable human view synthesis methods combine the pre-fitted 3D human meshes with image features to reach generalization, yet they are mainly designed to operate on single-human scenes. Another drawback is the reliance on multi-step optimization techniques for parametric pre-fitting of the 3D body models that suffer from misalignment with the images in sparse view settings causing hallucinations in synthesized views. In this work, we propose, GenLayNeRF, a generalizable layered scene representation for free-viewpoint rendering of multiple human subjects which requires no per-scene optimization and very sparse views as input. We divide the scene into multi-human layers anchored by the 3D body meshes. We then ensure pixel-level alignment of the body models with the input views through a novel end-to-end trainable module that carries out iterative parametric correction coupled with multi-view feature fusion to produce aligned 3D models. For NVS, we extract point-wise image-aligned and human-anchored features which are correlated and fused using self-attention and cross-attention modules. We augment low-level RGB values into the features with an attention-based RGB fusion module. To evaluate our approach, we construct two multi-human view synthesis datasets; DeepMultiSyn and ZJU-MultiHuman. The results indicate that our proposed approach outperforms generalizable and non-human per-scene NeRF methods while performing at par with layered per-scene methods without test time optimization.
BibTeX
@inproceedings{abdelkareem2023genlaynerf,
  author = {Youssef Abdelkareem and Shady Shehata and Fakhri Karray},
  title = {{GenLayNeRF: Generalizable Layered Representations with 3D Model Alignment for Human View Synthesis}},
  booktitle = {DAGM},
  year = {2023},
  url = {https://openreview.net/forum?id=ZRJ8jVp9UM},
}

Loading related content...

Figure