Abstract
To achieve true all-day vision for autonomous systems, perception must remain robust in conditions where standard RGB sensors fail, such as total darkness. This paper introduces the Multispectral Transfer Network (MTN), a novel unsupervised framework designed to estimate pixel-wise depth from a single thermal image. By leveraging geometric priors and chromaticity clues through a multi-task learning approach, MTN bridges the domain gap between thermal and visible spectrums. The architecture introduces a specialized "Interleaver" module to share features efficiently without direct layer sharing. Experimental results on a new multispectral stereo dataset demonstrate that MTN significantly outperforms existing baselines, particularly in challenging nighttime scenarios.
The Challenge of All-Day Perception
Standard depth estimation techniques are heavily reliant on RGB cameras, which are susceptible to failure in low-light environments like tunnels or nighttime streets. While active sensors like LiDAR can operate in the dark, they are often cost-prohibitive and provide sparse data. Thermal (Long-Wavelength Infrared) sensors offer a compelling alternative due to their lighting invariance, but they face two primary challenges: a lack of large-scale annotated depth datasets and a significant domain gap where thermal images lack the fine-grained texture and color information typically used for monocular depth inference.
Proposed Method: Multispectral Transfer Network (MTN)
The MTN framework addresses these challenges by using an unsupervised training strategy. During training, the model uses a calibrated RGB stereo pair to provide a geometric signal. The network learns to predict a disparity map from a thermal image, which is then used to warp a right-view RGB image to reconstruct the left-view. The reconstruction error serves as the training signal, eliminating the need for manual depth labels.
Key Architectural Innovations
- The Interleaver Module: A core component of the MTN is the Interleaver block. Inspired by gating mechanisms in recurrent neural networks, it uses global average pooling to intelligently share feature information between the primary depth estimation task and a secondary chromaticity (color) prediction task. This allows the model to learn richer geometric structures common to both domains.
- Multi-task Learning: By forcing the network to reconstruct the chromaticity (Cb and Cr channels) of the scene alongside depth, the encoder is incentivized to capture more detailed scene layouts.
- Adaptive Scaled Sigmoid ($S_{ass}$): To stabilize training and accurately represent objects close to the camera, the authors implemented a sigmoid function that adaptively scales its range as training progresses.
Datasets and Evaluation
The researchers introduced a Multispectral Stereo Dataset containing 7,383 sets of stereo RGB and thermal images captured across campus, residential, and urban environments. For evaluation, 3D LiDAR data was used as ground truth. To handle thermal contrast variations caused by heat shifts throughout the day, a data-driven photometric correction method was applied to normalize the thermal features.
Key Results and Insights
MTN demonstrates superior performance and generalization across various driving conditions:
- Accuracy Gains: In daytime tests, MTN improved the Root Mean Square Error (RMSE) by 22% at a 50m range compared to single-task networks.
- Nighttime Superiority: In total darkness, MTN's accuracy was three times higher than RGB-based methods, which fail entirely without external illumination.
- Visual Fidelity: Qualitatively, the depth maps produced by MTN are sharper and preserve object boundaries—such as cars and trees—more effectively than previous multi-task architectures like "Deep-shared" (DsMTN) or "Low-shared" (LsMTN) models.
Conclusion
The Multispectral Transfer Network provides a robust solution for all-day depth estimation by successfully transferring knowledge from the visible to the thermal domain. By utilizing an unsupervised multi-task framework and the novel Interleaver module, the system achieves high-resolution depth maps without the need for expensive LiDAR supervision. Future work will focus on refining loss functions to further reduce blurriness and improve long-range depth accuracy.
Summary updated