SpanConv: A New Convolution via Spanning Kernel Space for Lightweight Pansharpening

Zhi-Xuan Chen, Cheng Jin, Tian-Jing Zhang, Xiao Wu, Liang-Jian Deng

Abstract

Standard convolutional neural networks (CNNs) for pansharpening often suffer from high parameter redundancy and computational costs, making them difficult to deploy on resource-constrained satellite hardware. This paper introduces SpanConv, a lightweight convolution operation that treats the kernel space as a mathematical subspace. By learning a small set of "navigated kernels" (basis vectors) and spanning them with learnable coefficients, the authors develop LightNet. With only ~16,000 parameters, LightNet achieves state-of-the-art performance, outperforming models with millions of parameters while demonstrating superior generalization to unseen satellite data.

Introduction

Pansharpening is a critical task in remote sensing, involving the fusion of a high-resolution panchromatic (PAN) image with a low-resolution multispectral (MS) image. While deep learning has significantly improved fusion accuracy, traditional CNNs are often over-parameterized. This redundancy leads to three main issues: excessive computational cost for satellite deployment, potential overfitting to simulated training data, and a lack of interpretability in kernel weights. SpanConv addresses these challenges by redesigning the fundamental convolution operation through the lens of linear algebra.

The SpanConv Method

The core innovation, SpanConv (Spanning Convolution), replaces standard independent weight learning with a spanning strategy. Instead of learning every weight in a $C_{out} \times C_{in} \times k \times k$ kernel independently, SpanConv learns a small number ($N$) of "navigated kernels" ($\bar{K}_1, \bar{K}_2, \dots, \bar{K}_N$) which act as basis vectors.

For each input channel, the model learns specific coefficients ($\lambda$) to linearly combine these navigated kernels. This process "spans" a kernel space that effectively approximates standard convolution but with a fraction of the parameters.

The SpanConv process linearly combines navigated kernels (basis vectors) to generate the final convolution kernels.

The authors implemented this in LightNet, a streamlined architecture consisting of a Fusion Head (concatenating PAN and MS), an Extraction Belly (using "belly blocks" for feature extraction), and a Reconstruction Tail.

Analysis of Kernel Redundancy

A key question in designing SpanConv is determining the optimal number of navigated kernels ($N$). To investigate this, the authors performed Principal Component Analysis (PCA) on standard convolution kernels trained for pansharpening tasks.

Representation range of the first four principal components of standard convolution kernels, showing the dominance of the first two components.

As shown in Figure 2, the first two principal components capture the vast majority of the representation power, while subsequent components contribute significantly less. Based on this "sweet spot," the authors set $N=2$ for LightNet. This choice provides a balance between representation capacity and extreme efficiency, outperforming single-basis methods like Blueprint Separable Convolution (BSConv).

Experimental Results and Generalization

LightNet was evaluated on WorldView-3, QuickBird, and WorldView-2 datasets using Wald’s protocol.

Efficiency and Accuracy

LightNet utilizes only 16,000 parameters, making it roughly 100x smaller than benchmark models like DRPNN (1.8M) and BDPN (1.4M). Despite this reduction, it achieved top-tier results across standard metrics, including Spectral Angle Mapper (SAM) and relative dimensionless global error (ERGAS).

Robust Generalization

One of the most significant findings was LightNet's performance on unseen data. When trained on one satellite's data and tested on another (e.g., WorldView-2), LightNet outperformed much larger models. This suggests that the constrained kernel space of SpanConv acts as a powerful regularizer, preventing the model from overfitting to sensor-specific noise and allowing it to capture more fundamental physical features of the pansharpening task.

Conclusion

SpanConv offers a theoretically grounded approach to lightweight deep learning by viewing kernels as elements of a spanned subspace. By eliminating redundancy, LightNet achieves a remarkable balance of efficiency, accuracy, and robustness. While the optimal number of basis kernels may vary for other computer vision tasks, SpanConv serves as a highly effective "drop-in" replacement for standard convolutions in remote sensing applications.

Summary updated

Abstract

Standard convolution operations can effectively perform feature extraction and representation but result in high computational cost, largely due to the generation of the original convolution kernel corresponding to the channel dimension of the feature map, which will cause unnecessary redundancy. In this paper, we focus on kernel generation and present an interpretable span strategy, named SpanConv, for the effective construction of kernel space. Specifically, we first learn two navigated kernels with single channel as bases, then extend the two kernels by learnable coefficients, and finally span the two sets of kernels by their linear combination to construct the so-called SpanKernel. The proposed SpanConv is realized by replacing plain convolution kernel by SpanKernel. To verify the effectiveness of SpanConv, we design a simple network with SpanConv. Experiments demonstrate the proposed network significantly reduces parameters comparing with benchmark networks for remote sensing pansharpening, while achieving competitive performance and excellent generalization. Code is available at https://github.com/zhi-xuan-chen/IJCAI-2022 SpanConv.
BibTeX
@inproceedings{chen2022spanconv,
  author = {Zhi-Xuan Chen and Cheng Jin and Tian-Jing Zhang and Xiao Wu and Liang-Jian Deng},
  title = {{SpanConv: A New Convolution via Spanning Kernel Space for Lightweight Pansharpening}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/118},
}
Figure