Pan-Sharpening with Customized Transformer and Invertible Neural Network

Man Zhou, Jie Huang, Yanchi Fang, Xueyang Fu, Aiping Liu

Note on Retraction: As of July 2025, this research (originally presented at AAAI 2022) has been formally retracted by the authors and AAAI Press. Post-publication verification revealed significant data leakage, where test samples were inadvertently included in the training set. This resulted in artificially inflated performance metrics that are not reproducible under proper experimental conditions. The following summary describes the methodology as originally proposed.

Introduction

In satellite remote sensing, capturing high-resolution imagery often involves a trade-off between spatial and spectral detail. Satellites typically produce two distinct products: a high-resolution panchromatic (PAN) image, which is sharp but lacks color, and a low-resolution multispectral (MS) image, which provides rich color information but is blurry. Pan-sharpening is the computational process of fusing these two sources to generate a single high-resolution multispectral (HMS) image.

Traditional deep learning approaches, primarily based on Convolutional Neural Networks (CNNs), face two significant hurdles:

  1. Local Receptive Fields: CNNs are inherently local, making it difficult to capture long-range spatial dependencies across the entire image.
  2. Information Loss: Standard fusion techniques, such as simple concatenation, often discard critical spectral or spatial features during the integration process.

Architecture Overview

To address these limitations, the authors proposed a hybrid architecture that operates in two parallel streams: a local feature branch and a long-range feature branch. This dual-stream approach is designed to capture both fine-grained textures and global structures simultaneously.

The proposed pan-sharpening framework featuring local and long-range branches followed by INN-based fusion.

Customized Pan-Sharpening Transformer

The long-range branch utilizes a customized Transformer. Unlike standard self-attention mechanisms that operate within a single image, this model treats PAN and MS features as queries and keys. This cross-modality attention allows the network to "search" for high-frequency textures in the PAN image that correspond to specific spectral content in the MS image, effectively mapping sharp details onto the color information.

Invertible Neural Network (INN) Fusion

To mitigate information loss during the fusion stage, the framework employs an Invertible Neural Network (INN). Invertible architectures are mathematically designed to allow perfect reconstruction of the input from the output, theoretically ensuring a "lossless" fusion process.

Detailed structure of the Invertible Neural Network (INN) fusion module and its components, including the Half Instance Normalization (HIN) blocks.

The fusion module utilizes affine coupling layers and Half Instance Normalization (HIN) blocks. By maintaining invertibility, the network is constrained to preserve all relevant data from both the PAN and MS source images, reducing spectral distortion in the final output.

Experimental Methodology and Retraction Context

The study was originally evaluated using three major satellite datasets: WorldView-II, GaoFen2, and WorldView-III. The authors reported that their hybrid Transformer-INN model outperformed ten state-of-the-art algorithms across standard metrics, including PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity).

However, the subsequent discovery of data leakage—where the model was trained on data it was later tested on—invalidates these performance claims. While the architectural concepts of cross-modality Transformers and invertible fusion remain of theoretical interest to the remote sensing community, their practical efficacy compared to existing baselines requires rigorous re-validation using correctly partitioned datasets.

Conclusion

The proposed framework represents an ambitious attempt to combine the global modeling power of Transformers with the information-preserving characteristics of Invertible Neural Networks. While the reported state-of-the-art results were compromised by experimental errors, the methodology highlights a growing trend in remote sensing toward hybrid architectures that move beyond standard local convolutions.

Summary updated

Abstract

In remote sensing imaging systems, pan-sharpening is an important technique to obtain high-resolution multispectral images from a high-resolution panchromatic image and its corresponding low-resolution multispectral image. Owing to the powerful learning capability of convolution neural network (CNN), CNN-based methods have dominated this field. However, due to the limitation of the convolution operator, long-range spatial features are often not accurately obtained, thus limiting the overall performance. To this end, we propose a novel and effective method by exploiting a customized transformer architecture and information-lossless invertible neural module for long-range dependencies modeling and effective feature fusion in this paper. Specifically, the customized transformer formulates the PAN and MS features as queries and keys to encourage joint feature learning across two modalities while the designed invertible neural module enables effective feature fusion to generate the expected pan-sharpened results. To the best of our knowledge, this is the first attempt to introduce transformer and invertible neural network into pan-sharpening field. Extensive experiments over different kinds of satellite datasets demonstrate that our method outperforms state-of-the-art algorithms both visually and quantitatively with fewer parameters and flops. Further, the ablation experiments also prove the effectiveness of the proposed customized long-range transformer and effective invertible neural feature fusion module for pan-sharpening.
BibTeX
@inproceedings{zhou2022pan,
  author = {Man Zhou and Jie Huang and Yanchi Fang and Xueyang Fu and Aiping Liu},
  title = {{Pan-Sharpening with Customized Transformer and Invertible Neural Network}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i3.20267},
}
Figure