Abstract
Semantic Scene Completion (SSC) aims to reconstruct 3D geometry and semantic labels from a single RGB-D image. This task is challenging due to the inherent differences between photometric RGB data and geometric depth data. We present FFNet (Frequency Fusion Network), a novel approach that performs multi-modality fusion in the frequency domain. By utilizing the Discrete Cosine Transform (DCT) and learnable elliptical masks, FFNet effectively correlates color and depth features, achieving state-of-the-art performance on major benchmarks.
Introduction
Semantic Scene Completion is often considered the "holy grail" of 3D vision, requiring a model to predict the full 3D geometry of a scene—including occluded regions—from a single 2D viewpoint. The primary hurdle in this field is the effective fusion of RGB and depth information. While RGB images provide rich semantic textures, depth maps offer structural geometry. Standard spatial-domain fusion methods, such as simple concatenation or addition, often struggle to bridge the gap between these disparate data types, especially when depth data is noisy or uncertain.
Methodology
Overall Architecture
FFNet processes RGB and depth inputs through parallel 2D backbones. The features are then fused using the Frequency Fusion Module (FFM) before being projected into a 3D volume for final semantic completion.
Frequency Fusion Module (FFM)
The core innovation of FFNet is the FFM, which shifts the fusion process from the spatial domain to the frequency domain using the Discrete Cosine Transform (DCT). This module consists of three primary components:
- Frequency Attention (Elliptical Mask): The network employs a learnable elliptical mask to decompose features into low and high frequencies. By learning parameters such as the center, axes, and rotation of the ellipse, the model dynamically focuses on frequency bands where RGB and depth data are most complementary.
- Frequency Correlation: Leveraging the convolution theorem—where multiplication in the frequency domain corresponds to convolution in the spatial domain—the model multiplies RGB and depth features. This operation explicitly calculates patch similarity across the image, aligning texture patterns with geometric shapes.
- Feature Aggregation: The correlated features are transformed back to the spatial domain via Inverse DCT (IDCT) and used to recalibrate the original features before 3D projection.
2D-3D Pipeline
Following fusion, the enhanced 2D features are projected into a 3D voxel grid. A 3D subnetwork, utilizing Anisotropic Convolutional (AIC) modules and a 3D Sketch Hallucination module, fills in occluded voxels and assigns final semantic labels (e.g., chair, wall, floor).
Experimental Results
FFNet was evaluated on the NYU and NYU CAD datasets, setting new state-of-the-art benchmarks at the time of publication.
- NYU Dataset: FFNet achieved a 44.4% mIoU (mean Intersection over Union), significantly outperforming previous leading models like CCP (41.3%). Notably, FFNet reached these results using only NYU training data, without requiring massive synthetic pre-training datasets.
- NYU CAD Dataset: The model achieved a 57.4% mIoU, representing a substantial improvement over the previous best of 55.2%.
Qualitative analysis indicates that the FFM effectively "cleans up" features by reducing background noise in RGB data and recalibrating noisy depth features to align with semantic boundaries.
Conclusion
FFNet demonstrates that the frequency domain offers a powerful perspective for multi-modal fusion in 3D vision tasks. By using learnable elliptical masks to correlate RGB and depth features, the model overcomes the limitations of traditional spatial fusion. Future work may explore the application of this frequency-domain approach to other domains, such as autonomous driving and object tracking.
Summary updated