Siamese CNNs for RGB-LWIR Disparity Estimation

David-Alexandre Beaupré, Guillaume-Alexandre BilodeauWorkshops @ CVPR 2019 · DOI

Introduction

Stereo matching, the process of calculating disparity between corresponding pixels in a pair of images, is a fundamental task in 3D computer vision. While deep learning has achieved remarkable success in standard RGB-to-RGB stereo matching, cross-spectral matching—specifically between visible (RGB) and Long-Wavelength Infrared (LWIR) images—remains a significant challenge.

The primary obstacle is the "texture gap." Thermal images represent heat signatures and often lack the fine-grained intensity variations and color textures found in visible light. Historically, handcrafted descriptors like Mutual Information (MI) have outperformed neural networks in this domain due to their ability to find statistical correlations between disparate signal types. This paper explores a specialized Siamese Convolutional Neural Network (CNN) architecture designed to bridge this gap, focusing on human silhouette detection where thermal data is particularly advantageous.

Methodology

The authors propose a "dual-subnetwork" architecture that enforces geometric constraints through bidirectional matching. Instead of a single inference path, the model utilizes two identical Siamese branches, denoted as $N_L$ and $N_R$, which operate in parallel.

The dual-subnetwork architecture showing the parallel $N_L$ and $N_R$ branches and the final summation layer.

Siamese Branch Architecture

Each branch ($N_L$ and $N_R$) is designed to find features from one spectral domain within the other. Specifically, $N_L$ searches for an RGB reference patch within a larger LWIR search window, while $N_R$ performs the inverse operation.

Detailed view of a single Siamese branch, illustrating the feature extraction layers and the correlation mechanism.

As shown in the detailed branch view, each subnetwork consists of six convolutional layers, each followed by Batch Normalization and ReLU activation. These layers extract 64-dimensional feature vectors. A correlation layer then performs a dot-product between the reference feature vector and the search feature volume to produce a similarity score across the search space.

Left-Right Consistency and Loss

The "secret sauce" of this architecture is the enforcement of left-right consistency. The similarity outputs from both $N_L$ and $N_R$ are converted to log-probabilities and summed. This summation forces the model to be confident in both matching directions (RGB $\to$ LWIR and LWIR $\to$ RGB), effectively acting as a geometric constraint that filters out false positives.

For training, the model employs a cross-entropy loss with a smoothed target distribution. Rather than using a rigid "one-hot" vector for ground truth, the target probability is spread across a 3-pixel neighborhood, allowing the network to learn more gracefully in the presence of slight misalignments.

Datasets and Training

The study utilizes two primary datasets featuring human subjects in indoor environments: the LITIV Dataset (three video sequences with annotated disparity) and the St-Charles Dataset (used primarily for training). To compensate for the relatively small size of these datasets, the authors utilized heavy data augmentation, including mirroring and synthetic vertical shifts of ground truth points, which tripled the available training data.

Key Results and Insights

The Power of Parallelism

A key finding of the ablation study was that the dual-branch approach is essential for cross-spectral tasks. Combining $N_L$ and $N_R$ improved recall by 8% to 12% compared to using either branch in isolation. This suggests that the model can effectively leverage asymmetric information; if one domain (e.g., thermal) is featureless, the model can rely on the clearer "peak" provided by the other domain (e.g., RGB).

Performance Comparison

The proposed model achieved an average recall of 0.779 within a 3-pixel error margin. While it ranked second overall, it was only outperformed by Mutual Information (MI) when the latter used significantly larger window sizes (40x130 pixels vs. the CNN's 37x37). When compared at similar pixel counts (~1300 pixels), the CNN outperformed the handcrafted descriptor (0.779 vs 0.649), indicating superior information efficiency per pixel.

Conclusion and Future Work

This research demonstrates that a specialized Siamese architecture can match or exceed the performance of traditional statistical methods for cross-spectral disparity estimation. However, limitations remain. The model is currently optimized for human silhouettes and may struggle with inanimate objects at ambient temperatures. Additionally, the sensitivity of the model to specific training folds suggests a need for larger, more diverse cross-spectral datasets. Future work will focus on scaling the architecture and expanding its applicability beyond human-centric scenes.

Summary updated

BibTeX
@inproceedings{beaupre2019siamese,
  author = {David-Alexandre Beaupré and Guillaume-Alexandre Bilodeau},
  title = {{Siamese CNNs for RGB-LWIR Disparity Estimation}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  doi = {10.1109/cvprw.2019.00115},
}
Figure