Most computer vision textbooks describe stereo matching using a "standard" setup: two identical cameras, perfectly aligned, capturing the same scene at the same resolution. However, if you look at the back of a modern smartphone, you will see a different reality. You might find a high-resolution wide-angle lens paired with a lower-resolution telephoto lens, or even a thermal sensor alongside a standard RGB camera.
This shift toward asymmetric or unbalanced stereo creates a fundamental problem for traditional algorithms. When sensors differ in focal length, resolution, or modality, the assumption of pixel-to-pixel correspondence breaks down.
The Resolution Mismatch Problem
In a symmetric stereo pair, a feature in the left image is roughly the same size as the same feature in the right image. In the asymmetric arrays found in consumer hardware, this is rarely true. As noted by Pierluigi Zama Ramirez, Fabio Tosi, and their colleagues in the Booster: a Benchmark for Depth from Images of Specular and Transparent Surfaces project, mobile devices often pair a high-megapixel primary sensor with a much smaller secondary sensor.
This creates two primary hurdles:
- Aliasing and Ambiguity: A single pixel in a low-resolution sensor may represent an entire region in a high-resolution sensor. Standard matching costs (like census transforms or simple pixel differences) cannot handle this one-to-many relationship effectively.
- Scale Imbalance: As Önder Tuzcuoglu and the authors of XoFTR: Cross-modal Feature Matching Transformer point out, differences in focal length mean that objects appear at different scales across the two views, making rigid matching models ineffective.
Bridging Scales with Geometric Alignment
To solve the focal length discrepancy (e.g., matching a wide-angle view to a telephoto view), researchers have moved toward architectures that explicitly account for geometry. In TW-SMNet: Deep Multitask Learning of Tele-Wide Stereo Matching, Haoyu Ren and his team at Samsung use geometric operations like cropping and zero-padding to align feature maps before they enter a 3D convolutional network. This allows the model to compute disparities between a wide-angle view and the higher-resolution "subset" of the scene captured by the telephoto lens.
Similarly, Yunxiang Li and the authors of the Multi-Level Feature Fusion Network address "asymmetric degradation." They recognize that in real-world hardware, one lens is often blurrier or more distorted than the other. Their approach uses hybrid attention mechanisms to selectively fuse high-fidelity information from the "cleaner" view into the reconstruction of the "degraded" view.
Continuous Refinement and Feature-Metric Matching
If the sensors have different resolutions, at what resolution should the final depth map be? Downsampling the high-res image loses detail, while upsampling the low-res image introduces artifacts.
Fabio Tosi and Filippo Aleotti proposed a solution in Neural Disparity Refinement for Arbitrary Resolution Stereo. Instead of matching on a fixed grid, they use a continuous-coordinate neural refiner. This allows the system to predict disparities at any arbitrary resolution, effectively bridging the gap between mismatched sensors by treating the disparity map as a continuous function rather than a discrete matrix of pixels.
When ground-truth depth isn't available for training, Xihao Chen and Zhiwei Xiong suggest moving away from pixel-based consistency entirely. In Degradation-agnostic Correspondence from Resolution-asymmetric Stereo, they use a feature-metric loss. By matching deep features rather than raw pixels, the model becomes robust to the fact that the two images might have different levels of detail or noise.
Beyond RGB: Cross-Modal Correspondence
The most extreme version of asymmetric stereo occurs when the sensors capture different types of light. Fabio Tosi and Samuele Salti explored this in RGB-Multispectral Matching: Dataset, Learning Methodology, Evaluation, where they registered high-resolution RGB images with low-resolution multispectral data.
To train these models without perfect labels, they employed "proxy-supervision"—using a second RGB camera during training to generate high-quality labels that are then used to teach the model how to align the multispectral sensor. This line of work, along with the transformer-based matching in XoFTR, demonstrates that the same principles used for resolution-asymmetric stereo can enable complex sensor fusion, allowing devices to "see" across the electromagnetic spectrum with pixel-level precision.