GenLayNeRF: Generalizable Layered Representations with 3D Model Alignment for Multi-Human View Synthesis

Youssef Abdelkareem, Shady Shehata, Fakhri Karray

Abstract

Novel view synthesis (NVS) of multi-human scenes imposes challenges due to the complex inter-human occlusions. Layered representations handle the complexities by dividing the scene into multi-layered radiance fields, however, they are mainly constrained to per-scene optimization making them inefficient. Generalizable human view synthesis methods combine the pre-fitted 3D human meshes with image features to reach generalization, yet they are mainly designed to operate on single-human scenes. Another drawback is the reliance on multi-step optimization techniques for parametric pre-fitting of the 3D body models that suffer from misalignment with the images in sparse view settings causing hallucinations in synthesized views. In this work, we propose, GenLayNeRF, a generalizable layered scene representation for free-viewpoint rendering of multiple human subjects which requires no per-scene optimization and very sparse views as input. We divide the scene into multi-human layers anchored by the 3D body meshes. We then ensure pixel-level alignment of the body models with the input views through a novel end-to-end trainable module that carries out iterative parametric correction coupled with multi-view feature fusion to produce aligned 3D models. For NVS, we extract point-wise image-aligned and human-anchored features which are correlated and fused using self-attention and cross-attention modules. We augment low-level RGB values into the features with an attention-based RGB fusion module. To evaluate our approach, we construct two multi-human view synthesis datasets; DeepMultiSyn and ZJU-MultiHuman. The results indicate that our proposed approach outperforms generalizable and non-human per-scene NeRF methods while performing at par with layered per-scene methods without test time optimization.

Introduction

Synthesizing novel viewpoints of humans in motion—a task known as Novel View Synthesis (NVS)—is a cornerstone for the future of telepresence, virtual reality, and sports broadcasting. While Neural Radiance Fields (NeRF) have revolutionized the field, they face significant hurdles in multi-human scenarios. Inter-human occlusions, the requirement for lengthy per-scene optimization, and the frequent misalignment between 3D body models and input images often lead to visual artifacts and high computational costs.

To address these issues, we present GenLayNeRF, a generalizable layered scene representation. Unlike traditional methods that require hours of optimization for every new scene, GenLayNeRF operates in a zero-shot manner on unseen subjects and poses, leveraging sparse input views (typically 3-4) to produce high-fidelity renders without test-time training.

Methodology

GenLayNeRF’s architecture is built upon three primary pillars: layered representation, iterative alignment, and attention-based feature fusion.

Layered Scene Representation

The complexity of multi-human scenes is managed by decomposing the volume into distinct layers. Each human subject is treated as an independent radiance field anchored by a 3D SMPL (Skinned Multi-Person Linear) mesh. This layering allows the system to handle occlusions naturally; layers are rendered independently and then composited based on depth, preventing the "melting" or blurring of subjects often seen in global NeRF models when people interact closely.

Iterative Human-Image Alignment

A significant bottleneck in human-centric NVS is the reliance on pre-fitted 3D models. In sparse-view settings, these models often suffer from pixel-level misalignment. GenLayNeRF introduces an end-to-end trainable module that performs iterative parametric correction. By projecting the mesh onto input images and extracting local features, the model predicts a correction ($\Delta\Theta$) to the SMPL parameters. This ensures that the 3D anchor perfectly matches the 2D image evidence, drastically reducing the "ghosting" artifacts common in other models.

Attention-Aware and RGB Fusion

To synthesize a point in 3D space, the model must aggregate information from multiple camera views. Rather than simple averaging, GenLayNeRF employs Self-Attention and Cross-Attention mechanisms. This allows the network to dynamically weight features from different views, effectively ignoring cameras where a body part is occluded. Furthermore, an attention-based RGB fusion module injects low-level color information directly into the features, helping the model retain high-frequency details like clothing patterns and textures.

Experimental Setup and Datasets

Due to a lack of diverse multi-human NVS benchmarks, we introduced and adapted several datasets for evaluation:

  • DeepMultiSyn: Adapted from the DeepMultiCap dataset, featuring 2-3 humans in challenging scenarios like boxing and dancing.
  • ZJU-MultiHuman: A new dataset with 4 human subjects captured from 8 synchronized cameras.
  • ZJU-MoCap: A standard single-human dataset used to ensure the model maintains high performance on simpler, non-occluded scenes.

Key Results

GenLayNeRF demonstrates superior generalization capabilities compared to existing baselines. On the DeepMultiSyn dataset for "Unseen Models/Unseen Poses," the framework achieved a PSNR of 21.03, significantly outperforming other generalizable baselines such as NHP (19.51) and IBRNet (18.80).

Beyond image quality, the efficiency gains are substantial. While state-of-the-art per-scene methods (like L-NeRF) require approximately 144 hours of optimization per sequence, GenLayNeRF provides comparable or superior results with zero test-time optimization. Qualitative results show that the model maintains much better limb definition and facial features compared to baselines, which often struggle during complex inter-human occlusions.

Conclusion and Future Work

GenLayNeRF successfully bridges the gap between generalizable NVS and complex multi-human scene modeling. By integrating iterative alignment with a layered architecture, it overcomes the traditional pitfalls of mesh misalignment and occlusion.

However, limitations remain. The model currently relies on the SMPL mesh, which represents a "naked" human body, making it difficult to represent loose-fitting clothing like skirts or capes. Future work will focus on integrating more expressive models like SMPL-X to capture fine-grained details such as facial expressions and hand gestures, as well as developing robust generalizable background models for full-scene synthesis.

Summary updated

Abstract

Novel view synthesis (NVS) of multi-human scenes imposes challenges due to the complex inter-human occlusions. Layered representations handle the complexities by dividing the scene into multi-layered radiance fields, however, they are mainly constrained to per-scene optimization making them inefficient. Generalizable human view synthesis methods combine the pre-fitted 3D human meshes with image features to reach generalization, yet they are mainly designed to operate on single-human scenes. Another drawback is the reliance on multi-step optimization techniques for parametric pre-fitting of the 3D body models that suffer from misalignment with the images in sparse view settings causing hallucinations in synthesized views. In this work, we propose, GenLayNeRF, a generalizable layered scene representation for free-viewpoint rendering of multiple human subjects which requires no per-scene optimization and very sparse views as input. We divide the scene into multi-human layers anchored by the 3D body meshes. We then ensure pixel-level alignment of the body models with the input views through a novel end-to-end trainable module that carries out iterative parametric correction coupled with multi-view feature fusion to produce aligned 3D models. For NVS, we extract point-wise image-aligned and human-anchored features which are correlated and fused using self-attention and cross-attention modules. We augment low-level RGB values into the features with an attention-based RGB fusion module. To evaluate our approach, we construct two multi-human view synthesis datasets; DeepMultiSyn and ZJU-MultiHuman. The results indicate that our proposed approach outperforms generalizable and non-human per-scene NeRF methods while performing at par with layered per-scene methods without test time optimization.
BibTeX
@inproceedings{abdelkareem2023genlaynerf,
  author = {Youssef Abdelkareem and Shady Shehata and Fakhri Karray},
  title = {{GenLayNeRF: Generalizable Layered Representations with 3D Model Alignment for Multi-Human View Synthesis}},
  booktitle = {DAGM},
  year = {2023},
  url = {https://openreview.net/forum?id=ZRJ8jVp9UM},
}
Figure