SIPSA-Net: Shift-Invariant Pan Sharpening With Moving Object Alignment for Satellite Imagery

Jaehyup Lee, Soomin Seo, Munchurl KimCVPR 2021 · DOI

Abstract

Pan-sharpening is a critical process in satellite remote sensing, merging high-resolution panchromatic (PAN) images with low-resolution multi-spectral (MS) images. However, physical separation and temporal lags between sensors often lead to misalignments for moving objects, resulting in "comet tail" artifacts. This paper introduces SIPSA-Net, a deep-learning framework featuring a Feature Alignment Module (FAM) and a Shift-Invariant Spectral (SiS) loss. These innovations allow the network to align features in the abstract domain and ignore inherent sensor lags during training. Experimental results on the WorldView-3 dataset demonstrate that SIPSA-Net significantly reduces artifacts and improves spectral accuracy compared to existing state-of-the-art methods.

The Challenge: Misalignment and "Comet Tails"

In satellite imaging, the PAN and MS sensors are typically physically offset. While global registration techniques can align static terrain, moving objects like vehicles appear at different coordinates in the PAN and MS captures. Traditional deep learning models, which rely on pixel-wise loss functions against a "ground truth" MS image, struggle with this discrepancy. The result is often a "comet tail" artifact—a color smear trailing behind a moving object—or double-edge blurring.

Methodology: SIPSA-Net Architecture

SIPSA-Net addresses these issues through a two-stage process: feature alignment followed by sharpening.

The SIPSA-Net architecture featuring the Feature Alignment Module (FAM) and the Pan-Sharpening Module (PSM).

1. Feature Alignment Module (FAM)

Rather than aligning raw pixels, which is prone to error due to resolution differences, the FAM operates in the feature domain. It utilizes Pixel-wise Probabilistic Alignment Convolution (PWPAC) to learn a probability map of offsets. This allows the network to predict where a color feature from the MS image should be shifted to match the high-resolution structural details of the PAN image.

2. Pan-Sharpening Module (PSM)

The PSM performs the final fusion of the aligned MS features and the PAN image. It employs a series of Residual Blocks (ResBlocks) and a Pixel-Shuffle layer to upscale the image, ensuring that the final output retains the sharp edges of the PAN data while incorporating the correct spectral information.

3. Shift-Invariant Spectral (SiS) Loss

A key innovation is the SiS loss function. Standard loss functions penalize any deviation from the input MS image. Since the input MS image is often misaligned for moving objects, standard training forces the model to produce blurry, misaligned results. The SiS loss calculates the difference between the output and multiple shifted versions of the MS input, selecting the minimum error. This "shift-invariant" approach allows the model to focus on structural alignment without being penalized for the sensor's inherent temporal lag.

Experimental Results

The model was evaluated using the WorldView-3 dataset, which includes 8 spectral bands and a 4x resolution difference between MS and PAN images.

Qualitative comparison on a highway scene. SIPSA-Net (j) eliminates the color smearing seen in other methods like PanNet (c) and DSen2 (h).

Quantitative Performance

SIPSA-Net achieved superior results across standard metrics. In full-scale validation, it reached a Joint Quality Measure (JQM) of 0.934, significantly higher than the next best method, PanNet-S3 (0.827). Ablation studies confirmed that the FAM is critical; removing it caused the JQM to drop to 0.576, highlighting that alignment is the primary bottleneck in modern pan-sharpening.

Qualitative Superiority

As seen in the highway comparison, SIPSA-Net produces remarkably sharp images of moving vehicles. While other methods like PanNet and DSen2 exhibit significant "comet tails" or color bleeding, SIPSA-Net successfully contains the color within the object boundaries defined by the high-resolution PAN image.

Conclusion

SIPSA-Net represents a significant advancement in satellite image processing by explicitly modeling and correcting for moving object misalignment. By shifting the alignment task to the feature domain and employing a shift-invariant loss, the framework overcomes the limitations of traditional pixel-wise training. Future work may involve optimizing the search window for the FAM to handle even higher-velocity objects or varying sensor altitudes.

Summary updated

BibTeX
@inproceedings{lee2021sipsa,
  author = {Jaehyup Lee and Soomin Seo and Munchurl Kim},
  title = {{SIPSA-Net: Shift-Invariant Pan Sharpening With Moving Object Alignment for Satellite Imagery}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.01003},
}
Figure