Abstract
Generative Adversarial Networks (GANs) and their variants have achieved remarkable success on natural images. However, their performance degrades when applied to remote sensing (RS) images, and the discriminator often suffers from the overfitting problem. In this paper, we examine the differences between natural and RS images and find that the intrinsic dimensions of RS images are much lower than those of natural images. As the discriminator is more susceptible to overfitting on data with lower intrinsic dimension, it focuses excessively on local characteristics of RS training data and disregards the overall structure of the distribution, leading to a faulty generation model. In response, we propose a novel approach that leverages the real data manifold to constrain the discriminator and enhance the model performance. Specifically, we introduce a learnable information-theoretic measure to capture the real data manifold. Building upon this measure, we propose manifold alignment regularization, which mitigates the discriminator's overfitting and improves the quality of generated samples. Moreover, we establish a unified GAN framework for manifold alignment, applicable to both supervised and unsupervised RS image generation tasks.
Introduction: The Remote Sensing Challenge
While GANs have revolutionized image generation for faces and natural scenery, their performance on satellite and aerial imagery (Remote Sensing) has been underwhelming. A persistent gap exists: GANs trained on RS data tend to overfit and diverge much sooner than those trained on natural datasets like FFHQ.
The root cause lies in the intrinsic dimension of the data. Research indicates that RS images often have a lower intrinsic dimension than natural images. In GAN training, a lower-dimensional manifold makes it easier for the discriminator to perfectly separate real samples from generated ones by memorizing local features. Once the discriminator overfits, its feedback to the generator becomes uninformative, leading to the "early divergence" and poor image quality common in the field.
Manifold Constraint Regularization (MCR)
The core contribution of this work is Manifold Constraint Regularization (MCR). Instead of the discriminator merely distinguishing "real vs. fake" pixels, MCR imposes a geometric constraint based on the underlying data manifold.
Key technical components include:
- Compactness Measure: The authors use the trace of the feature covariance matrix ($Tr(ZZ^T)$) as a proxy for distribution compactness, which is theoretically linked to information-theoretic rate distortion.
- Manifold Evaluation Function: A loss term encourages features from different submanifolds (e.g., different terrain types) to be uncorrelated, while maintaining coherence within the same submanifold.
- Unsupervised Relationship Learning: For unlabeled data, a three-layer MLP automatically learns relationships between data points in the feature space.
- Loss Integration: This MCR term is added to the standard GAN objective, pushing the generator to produce an output manifold that aligns with the global structure of the real data rather than just mimicking individual samples.
Experimental Results
The proposed approach was validated on major RS benchmarks, including NWPU-RESISC45 (NWPU), PatternNet (PN), and UC Merced Land Use (UCLand), comparing them against natural datasets like FFHQ and TinyImageNet.
Performance Gains
MCR effectively mitigates the early divergence problem:
- Quantitative Improvement: The method achieved a 3.13% improvement in FID scores across multiple datasets. On the NWPU dataset, it reached an FID of 9.88, outperforming state-of-the-art regularization methods like MoCA and LeCam.
- Downstream Utility: When used for data augmentation in classification tasks, MCR-generated images led to a 3.76% increase in accuracy, proving the generated images capture authentic features of the RS domain.
- Qualitative Fidelity: Visual inspections show that MCR-generated images have more regular shapes and well-defined object boundaries (like buildings and parking lots) compared to standard GAN outputs, which often appear distorted.
Conclusion
By focusing on manifold alignment rather than simple sample matching, this framework provides a robust solution to discriminator overfitting in RS image generation. MCR is model-agnostic and has been successfully integrated into architectures like StyleGAN2 and BigGAN without requiring major structural changes. Future work will explore extending this manifold-based approach to multi-spectral data and other generative paradigms like Diffusion Models.
Summary updated