Abstract
Pan-sharpening is a critical process in satellite remote sensing, merging high-resolution panchromatic (PAN) images with low-resolution multi-spectral (MS) images. However, physical separation and temporal lags between sensors often lead to misalignments for moving objects, resulting in "comet tail" artifacts. This paper introduces SIPSA-Net, a deep-learning framework featuring a Feature Alignment Module (FAM) and a Shift-Invariant Spectral (SiS) loss. These innovations allow the network to align features in the abstract domain and ignore inherent sensor lags during training. Experimental results on the WorldView-3 dataset demonstrate that SIPSA-Net significantly reduces artifacts and improves spectral accuracy compared to existing state-of-the-art methods.
The Challenge: Misalignment and "Comet Tails"
In satellite imaging, the PAN and MS sensors are typically physically offset. While global registration techniques can align static terrain, moving objects like vehicles appear at different coordinates in the PAN and MS captures. Traditional deep learning models, which rely on pixel-wise loss functions against a "ground truth" MS image, struggle with this discrepancy. The result is often a "comet tail" artifact—a color smear trailing behind a moving object—or double-edge blurring.
Methodology: SIPSA-Net Architecture
SIPSA-Net addresses these issues through a two-stage process: feature alignment followed by sharpening.
1. Feature Alignment Module (FAM)
Rather than aligning raw pixels, which is prone to error due to resolution differences, the FAM operates in the feature domain. It utilizes Pixel-wise Probabilistic Alignment Convolution (PWPAC) to learn a probability map of offsets. This allows the network to predict where a color feature from the MS image should be shifted to match the high-resolution structural details of the PAN image.
2. Pan-Sharpening Module (PSM)
The PSM performs the final fusion of the aligned MS features and the PAN image. It employs a series of Residual Blocks (ResBlocks) and a Pixel-Shuffle layer to upscale the image, ensuring that the final output retains the sharp edges of the PAN data while incorporating the correct spectral information.
3. Shift-Invariant Spectral (SiS) Loss
A key innovation is the SiS loss function. Standard loss functions penalize any deviation from the input MS image. Since the input MS image is often misaligned for moving objects, standard training forces the model to produce blurry, misaligned results. The SiS loss calculates the difference between the output and multiple shifted versions of the MS input, selecting the minimum error. This "shift-invariant" approach allows the model to focus on structural alignment without being penalized for the sensor's inherent temporal lag.
Experimental Results
The model was evaluated using the WorldView-3 dataset, which includes 8 spectral bands and a 4x resolution difference between MS and PAN images.
Quantitative Performance
SIPSA-Net achieved superior results across standard metrics. In full-scale validation, it reached a Joint Quality Measure (JQM) of 0.934, significantly higher than the next best method, PanNet-S3 (0.827). Ablation studies confirmed that the FAM is critical; removing it caused the JQM to drop to 0.576, highlighting that alignment is the primary bottleneck in modern pan-sharpening.
Qualitative Superiority
As seen in the highway comparison, SIPSA-Net produces remarkably sharp images of moving vehicles. While other methods like PanNet and DSen2 exhibit significant "comet tails" or color bleeding, SIPSA-Net successfully contains the color within the object boundaries defined by the high-resolution PAN image.
Conclusion
SIPSA-Net represents a significant advancement in satellite image processing by explicitly modeling and correcting for moving object misalignment. By shifting the alignment task to the feature domain and employing a shift-invariant loss, the framework overcomes the limitations of traditional pixel-wise training. Future work may involve optimizing the search window for the FAM to handle even higher-velocity objects or varying sensor altitudes.
Summary updated