Semi-Supervised Image Stitching from Unstructured Camera Arrays

Erman Nghonda Tchinda, Maximillian Kealoha Panoff, Danielle Tchuinkou Kwadjo, Christophe BobdaSensors 2023 · DOI · OpenReview

TL;DR

This paper introduces SandFall, a novel deep learning approach for stitching images from unstructured camera arrays, overcoming limitations of traditional pairwise methods. By representing images in a compressed, fixed-size 3D volume, it achieves significantly faster stitching times and better scalability for systems with many cameras, while maintaining competitive image quality.

What Problem Does This Paper Address?

Image stitching, the process of combining multiple images to create a wider field of view, has numerous applications. However, existing methods, particularly those relying on sequential pairwise stitching, struggle with unstructured camera arrays (where cameras aren't neatly arranged). These methods are prone to error propagation, leading to artifacts like broken objects and ghosting. Furthermore, they often require significant computational resources and don't scale well with an increasing number of cameras. The challenge is to efficiently stitch images from a large, unorganized set of cameras into a seamless panorama without sacrificing quality or speed.

Methods

The core of this work lies in two main innovations:

  • SandFall Algorithm: This algorithm addresses the challenge of handling an arbitrary number of input images by creating a compressed, fixed-size 3D "blob." It takes warped input images and layers them into a volume, then "drops" pixels through this volume, effectively prioritizing central pixels and reducing data loss. This results in a consistent input size for the neural network, regardless of the number of cameras.
  • Deep Learning Stitching Network: A custom convolutional neural network (CNN) is employed to process the SandFall blob. This network is trained using a semi-supervised learning approach. This means it leverages a small amount of labeled data (ground truth panoramas created by traditional methods) and a larger amount of unlabeled data. This approach significantly reduces the reliance on extensive, manually annotated datasets, which are difficult to obtain for this task. The network's objective function combines several loss components, including content loss, reconstruction loss, total variation, and per-pixel loss, to ensure both structural integrity and visual quality.

Datasets

The authors utilized two primary sources for their experiments:

  • Publicly Available Dataset: A large, unlabeled dataset from Walt Disney was used, notable for its use of multiple cameras in an unstructured manner.
  • Custom Dataset: The authors created their own dataset by recording synchronized videos from multiple Raspberry Pi 3Bs equipped with Sony IMX219 sensors. These videos were converted into frame-by-frame image sequences, capturing both indoor and outdoor scenes.

Key Insights and Ideas

  • Scalability through Fixed-Size Representation: The SandFall algorithm's ability to compress an arbitrary number of camera inputs into a fixed-size representation is a key enabler for scalability. This bypasses the quadratic complexity issues of pairwise methods.
  • Semi-Supervised Learning for Data Scarcity: The adoption of semi-supervised learning is crucial. It acknowledges the difficulty of obtaining large, labeled datasets for complex tasks like stitching from unstructured arrays, making the approach more practical.
  • Feature-Based Metrics for Quality: The paper highlights the inadequacy of traditional pixel-based metrics (like MSE, SSIM) for evaluating stitched images when ground truth is unreliable. Instead, they advocate for feature-based metrics (LPIPS, FID, IS) that better correlate with human perception of image quality and structural integrity.
  • Intrinsic Artifact Mitigation: The deep learning model inherently handles common image artifacts like noise and JPEG compression, which often require separate post-processing steps in traditional methods.

Key Results

  • Speed: The proposed method achieves significantly faster stitching times. On average, it requires approximately 1/7th the time of traditional multi-band and seam-based stitching methods, even when run on a CPU (outperforming them even without GPU acceleration).
  • Performance: While not always outperforming traditional methods on all metrics, the proposed approach shows competitive results, particularly excelling in Fréchet Inception Distance (FID). It also demonstrates better handling of "broken objects" and seamless blending compared to existing methods.
  • Scalability: The method's performance is largely independent of the number of cameras, making it highly scalable for systems with many cameras.

Limitations and Future Work

  • Data Loss in SandFall: While SandFall minimizes data loss, it acknowledges that some loss can occur, especially with increasing overlap. The current implementation lacks specific criteria for selecting which values to preserve, leaving this as a direction for future work.
  • Subjectivity in Qualitative Comparisons: The authors admit that qualitative comparisons, while showing fewer artifacts, are inherently subjective.
  • Further Optimization of Loss Function: While the current loss function is effective, further exploration into optimizing its components and potentially incorporating more advanced metrics could lead to even better results.
  • Real-time Performance: While significantly faster, achieving true real-time performance for very high-resolution or extremely large camera arrays might require further optimization and GPU acceleration.

Summary updated

Abstract

Image stitching involves combining multiple images of the same scene captured from different viewpoints into a single image with an expanded field of view. While this technique has various applications in computer vision, traditional methods rely on the successive stitching of image pairs taken from multiple cameras. While this approach is effective for organized camera arrays, it can pose challenges for unstructured ones, especially when handling scene overlaps. This paper presents a deep learning-based approach for stitching images from large unstructured camera sets covering complex scenes. Our method processes images concurrently by using the SandFall algorithm to transform data from multiple cameras into a reduced fixed array, thereby minimizing data loss. A customized convolutional neural network then processes these data to produce the final image. By stitching images simultaneously, our method avoids the potential cascading errors seen in sequential pairwise stitching while offering improved time efficiency. In addition, we detail an unsupervised training method for the network utilizing metrics from Generative Adversarial Networks supplemented with supervised learning. Our testing revealed that the proposed approach operates in roughly ∼1/7th the time of many traditional methods on both CPU and GPU platforms, achieving results consistent with established methods.
BibTeX
@article{tchinda2023semi,
  author = {Erman Nghonda Tchinda and Maximillian Kealoha Panoff and Danielle Tchuinkou Kwadjo and Christophe Bobda},
  title = {{Semi-Supervised Image Stitching from Unstructured Camera Arrays}},
  journal = {Sensors},
  year = {2023},
  doi = {10.3390/s23239481},
}
Figure