LGPConv: Learnable Gaussian Perturbation Convolution for Lightweight Pansharpening

Chen-Yu Zhao, Tian-Jing Zhang, Ran Ran, Zhi-Xuan Chen, Liang-Jian DengIJCAI 2023 · DOI

Abstract

LGPConv introduces a lightweight convolution operation that utilizes learnable Gaussian perturbations to overcome the redundancy typically found in standard neural network kernels. By combining a "premier" base kernel with controlled random noise, the proposed LGPConv-Net achieves state-of-the-art (SOTA) pansharpening performance with a remarkably small footprint of only 27,000 parameters and 0.112 GFLOPS.

Introduction

Pansharpening is the critical process of fusing a low-resolution multispectral (LR-MS) image with a high-resolution panchromatic (PAN) image to produce a high-resolution multispectral (HR-MS) output. While Deep Learning (DL) has significantly advanced this field, current models face two primary hurdles: high computational costs and kernel redundancy. Standard Convolutional Neural Networks (CNNs) often require massive parameter counts, making them unsuitable for resource-constrained satellite hardware. Furthermore, many parameters in standard convolutions are "wasted" because kernels across different channels are often highly correlated.

Existing lightweight solutions, such as Blueprint Separable Convolution (BSConv), reduce parameter counts but suffer from limited representation capability because their kernels are strictly linear transformations of a base blueprint. LGPConv is designed to bridge this gap by restoring representational power through stochastic perturbation.

Methodology: Learnable Gaussian Perturbation

The core innovation, LGPConv (Learnable Gaussian Perturbation Convolution), operates in two distinct stages to maximize efficiency without sacrificing performance:

  1. The Premier Kernel: Inspired by the efficiency of BSConv, the model learns a "blueprint" kernel and a set of weights to capture correlations between channels. This significantly reduces the initial parameter count compared to standard convolutions.
  2. Gaussian Perturbation: To overcome the linearity constraints of the premier kernel, the authors inject learnable Gaussian noise perturbations ($\eta$). This randomness allows the kernel to span a much wider "representation space" without adding a significant number of learnable parameters.
Comparison of Standard Convolution, BSConv, and the proposed LGPConv mechanism.

The authors integrate LGPConv into a streamlined architecture called LGPConv-Net, which consists of a feature extractor, a deep inference block utilizing ResNet-like structures, and a fusion reconstruction layer. Mathematical proof using Lipschitz conditions demonstrates that training with these perturbations is equivalent to a form of implicit regularization, ensuring network stability and reliable convergence despite the injection of randomness.

Representation Space Analysis

A key insight of this work is the distinction between subspace and full-space representation. Using Principal Component Analysis (PCA), the authors demonstrate that standard kernels are inherently redundant. While lightweight models like BSConv are often trapped in a restricted linear "subspace," LGPConv uses perturbation to "fill" the kernel space.

Visualization of representation ranges and vector spaces for Standard, BSConv, and LGPConv methods.

As shown in the representation analysis, LGPConv achieves a representation range comparable to or exceeding standard convolutions, whereas BSConv remains highly constrained. This allows LGPConv-Net to regain the representational power of much heavier models while maintaining a lightweight profile.

Experimental Results

The model was evaluated on three major satellite datasets: WorldView-3 (WV-3), Gaofen-2 (GF-2), and QuickBird (QB). Experiments included both reduced-resolution (simulated) and full-resolution (real-world) testing.

LGPConv-Net outperformed existing SOTA methods across nearly all metrics:

  • Parameter Efficiency: It utilizes only 27K parameters, making it 3x to 4x smaller than high-performing models like FusionNet (79K) and PNN (104K).
  • Computational Speed: It achieves the lowest GFLOPS (0.112) among all compared deep learning methods.
  • Accuracy: On the WV-3 dataset, it achieved top scores in Spectral Angle Mapper (SAM), ERGAS, and Spatial Correlation Coefficient (SCC), indicating superior preservation of both spectral (color) and spatial detail.

Conclusion and Future Work

LGPConv-Net demonstrates that embracing controlled noise can be a powerful tool for model compression. By using Gaussian perturbations as a form of implicit regularization, the model achieves high fidelity with minimal hardware requirements. However, the authors note that the model currently shows limited generalization when moving between different types of sensors, as the perturbations are sensitive to specific training data distributions. Future research will focus on making the perturbation mechanism more adaptive to diverse satellite sensors to improve cross-dataset robustness.

Summary updated

BibTeX
@inproceedings{zhao2023lgpconv,
  author = {Chen-Yu Zhao and Tian-Jing Zhang and Ran Ran and Zhi-Xuan Chen and Liang-Jian Deng},
  title = {{LGPConv: Learnable Gaussian Perturbation Convolution for Lightweight Pansharpening}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/517},
}
Figure