QUADify: Extracting Meshes with Pixel-Level Details and Materials from Images

Maximilian Frühauf, Hayko Riemenschneider, Markus Gross, Christopher Schroers

Abstract

QUADify is an end-to-end framework that reconstructs high-quality 3D objects from 2D images directly into a quad-dominant mesh format. By making the re-meshing process differentiable, the system produces clean, artist-friendly geometry with pixel-level details and decomposed materials. Unlike traditional methods that produce irregular triangle soups, QUADify integrates quad-extraction into the optimization loop, ensuring that the underlying geometry is optimized specifically for high-quality topology and professional rendering pipelines.

Introduction

Most modern 3D reconstruction techniques, such as NeRFs or photogrammetry, produce either implicit neural volumes or irregular triangle meshes. While these representations can achieve high visual fidelity, they are often unsuitable for professional industry workflows. Artists prefer quad meshes for their predictable behavior during sculpting, animation, and subdivision. Traditionally, converting a raw 3D scan into a clean quad mesh is a manual, non-differentiable post-processing step. QUADify addresses this by making the quad-extraction process part of the differentiable optimization loop, allowing the system to learn a shape that favors clean topology.

Methods

The authors bridge the gap between implicit neural representations and explicit geometric editing through several key innovations:

  1. Differentiable Re-meshing: The system represents shape using a Signed Distance Field (SDF). It learns "orientation" ($O$) and "position" ($P$) fields across the surface via two small neural networks (o-MLP and p-MLP). These fields guide the placement of quad faces.
Orientation and position fields guide the placement of quad faces on the surface.
  1. The Stop-Gradient Trick: To handle the non-differentiable nature of iterative smoothing, the authors use a technique similar to Vector Quantization. They skip the gradients of the iterative smoothing process and instead "copy" the gradients from the final quad mesh directly back to the initial predicted fields. This allows the SDF to learn a shape that specifically favors a clean quad layout.
  2. Subdivision and Displacement: Once a base quad mesh is formed, the system applies Catmull-Clark subdivision, the industry standard for smoothing meshes. To capture fine details like wrinkles or pores, it learns a displacement field that shifts vertices at the pixel level, rather than relying solely on shading tricks like normal maps.
  3. Inverse Rendering: The entire pipeline is supervised by an image-space loss. The model renders its current mesh, compares it to the original photos, and back-propagates the error to update the geometry, materials (diffuse and specular), and lighting.

Key Results

QUADify was evaluated on the NeRF Synthetic, Myles, and NeRD datasets. It demonstrated superior mesh quality, achieving a 0.00% outlier percentage for face aspect ratios on the Myles dataset, compared to ~12-16% for standard Marching Cubes. Geometrically, it outperformed baselines like Nvdiffrec, achieving a Chamfer Distance of 4.63 compared to Nvdiffrec's 7.8.

Comparison of QUADify's editable mesh, fine details, and materials against Nvdiffrec.

The method produces competitive render quality while providing a mesh that is significantly easier to edit and animate than traditional triangle-based reconstructions. By prioritizing quad-dominant topology, it provides a stable canvas for displacement mapping and material decomposition.

Limitations and Future Work

While QUADify represents a significant step forward, it faces challenges in reconstructing deep concave regions where the MLP-based SDF can get stuck in local minima. Additionally, the current implementation initializes the SDF as a sphere, which can struggle with complex topological changes if foreground masks are inconsistent. Future work will focus on moving toward more flexible representations that can handle arbitrary topology and improving the reconstruction of highly complex real-world scenes.

Summary updated

Abstract

Despite exciting progress in automatic 3D reconstruction from images, excessive and irregular triangular faces in the resulting meshes still constitute a significant challenge when it comes to adoption in practical artist work-flows. Therefore, we propose a method to extract regular quad-dominant meshes from posed images. More specifically, we generate a high-quality 3D model through de-composition into an easily editable quad-dominant mesh with pixel-level details such as displacement, materials, and lighting. To enable end-to-end learning of shape and quad topology, we QUADify a neural implicit representation using our novel differentiable re-meshing objective. Distinct from previous work, our method exploits artifact-free Catmull-Clark subdivision combined with vertex displacement to extract pixel-level details linked to the base geom-etry. Finally, we apply differentiable rendering techniques for material and lighting decomposition to optimize for image reconstruction. Our experiments show the benefits of end-to-end re-meshing and that our method yields state-of-the-art geometric accuracy while providing lightweight meshes with displacements and textures that are directly compatible with professional renderers and game engines.
BibTeX
@inproceedings{fruhauf2024quadify,
  author = {Maximilian Frühauf and Hayko Riemenschneider and Markus Gross and Christopher Schroers},
  title = {{QUADify: Extracting Meshes with Pixel-Level Details and Materials from Images}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00446},
}
Figure