Abstract
High-fidelity 3D surface reconstruction is a computationally intensive task, often hindered by the irregular nature of 3D data formats. This paper presents MNSRNet, a multimodal transformer network designed for 3D surface super-resolution in the 2D normal domain. By integrating texture, depth, and normal information, MNSRNet overcomes the limitations of single-modality approaches. The network employs a Multimodal Pre-processing Stage (MPS) to generate bridge normal maps, a Cross-modality Transformer Alignment (cmTA) module for feature synchronization, and a Cross-modality Affine Fusion (cmAF) mechanism for adaptive feature integration. Evaluated on the new large-scale WPS dataset and existing benchmarks, MNSRNet demonstrates superior performance in recovering fine-grained surface details compared to state-of-the-art 3D and 2D super-resolution methods.
Introduction
Obtaining high-quality 3D surface details is traditionally expensive and computationally taxing. While 3D data can be represented as voxels, point clouds, or meshes, these formats are often "unstructured" or "irregular," making standard deep learning up-sampling difficult. Previous attempts to solve this in the 2D domain by up-sampling normal maps—2D images where pixels represent surface orientation—often relied on Single Image Super-Resolution (SISR). However, SISR typically only uses one modality (the normal map itself), ignoring the rich geometric cues hidden in texture photographs and depth maps.
The challenge lies in effectively aligning and fusing these different types of data (modalities) which have vastly different mathematical properties. MNSRNet addresses this by operating in the 2D normal domain, leveraging mature image processing architectures while avoiding the computational "curse of dimensionality" found in 3D voxels.
Methodology
MNSRNet consists of three innovative stages designed to handle multimodal data effectively:
1. Multimodal Pre-processing Stage (MPS)
To handle the "noise" in raw photos, such as shadows or specular reflections, the MPS extracts a "hierarchical texture" (darkest, lightest, and average views). It also generates "bridge normal maps"—intermediate representations that act as a common language to link depth and texture data to the normal domain. Specifically, it produces a Shape Normal ($N_s$) from depth and a Texture Normal ($N_t$) from RGB images.
2. Cross-modality Transformer Alignment (cmTA)
This module uses the self-attention mechanism of Transformers to align the side modalities (texture and depth) with the main normal map. By using the "bridge normals" as anchors, the network learns how to project high-frequency texture details onto the geometric surface. Transformers excel here because they capture global correlations, which is essential for aligning textures captured from different perspectives with a geometric structure.
3. Cross-modality Affine Fusion (cmAF)
Instead of simple concatenation, MNSRNet uses an affine transform mechanism (scaling and shifting) to inject side-modality features into the main super-resolution backbone. This allows the network to adaptively decide the influence of texture or depth on specific parts of the surface.
Datasets and Evaluation
The study utilizes two established benchmarks and introduces a significant new one:
- DiLiGenT: A standard photometric stereo dataset.
- Gourd & Apple: A small dataset with 3 objects.
- WPS (Wonderful Photometric Stereo): A new, large-scale dataset containing 400 different objects (butterfly wings, leaves, handicrafts, etc.) captured under 18 predefined lighting conditions.
MNSRNet was compared against nine competitive schemes, including mesh-based (Catmull-Clark), point-cloud-based (PU-GCN), and state-of-the-art image SR methods (RCAN, IPT).
Key Results
MNSRNet achieved quantitative superiority, reaching a PSNR of 28.76 on the $\times2$ scale, outperforming the next best multimodal method (3DASR) significantly. It also achieved the lowest Mean Angular Error (MAE) and Mean Relative Depth Error (MRDE). Qualitatively, the model recovers fine-grained details—such as the porous texture of stone or delicate leaf veins—that other methods either blur or distort with artifacts.
Limitations and Future Work
The method relies on photometric stereo data, which is harder to achieve in "in-the-wild" scenarios compared to single-shot photography. Additionally, Transformer blocks are memory-intensive, requiring adaptive cropping for fixed input sizes. Future work may explore achieving similar results with fewer input lightings or moving toward "uncalibrated" photometric stereo.
Summary updated