Locality-Preserving Directions for Interpreting the Latent Space of Satellite Image GANs

Georgia Kourmouli, Nikos Kostagiolas, Mihalis A. Nicolaou, Yannis Panagakis

TL;DR

Researchers have developed a new "locality-aware" method to navigate the latent space of Generative Adversarial Networks (GANs) specifically for satellite imagery. By replacing standard global variance techniques (like PCA) with Locality Preserving Projections (LPP), the method discovers semantic directions—like urbanization or vegetation growth—that produce more realistic, artifact-free synthetic images, significantly boosting the performance of downstream classification tasks.

What Problem Does This Paper Address?

Satellite imagery is notoriously difficult for generative models due to its extreme spatial and spectral variability. While GANs can generate high-quality synthetic satellite images, "steering" these models to produce specific features (e.g., "add more buildings here") usually involves analyzing the GAN's latent space using Principal Component Analysis (PCA).

The problem? PCA focuses on global variance—it looks for the directions of maximum change across the entire dataset. In the complex, high-dimensional manifold of satellite data, these global directions often ignore local structures. This leads to "edits" that introduce unrealistic artifacts or completely change the class of the image (e.g., turning a forest into a desert instead of just adding a few more trees). The authors argue that for satellite data, preserving locality—the relationships between neighboring points in the data—is key to meaningful image manipulation.

Methods

The authors propose using Locality Preserving Projections (LPP) to interpret the latent space of a pre-trained SWAGAN (a wavelet-based GAN known for capturing fine spatial details).

  1. Weight Space Analysis: Instead of looking at the covariance of the GAN's projection layer weights (as PCA does), they build an adjacency graph. This graph connects each weight vector to its $k$-nearest neighbors.
  2. LPP Factorization: They solve a generalized eigenvalue problem to find a linear mapping that keeps connected points in the graph as close as possible in the reduced space.
  3. Semantic Discovery: This process yields "interpretable directions" in the latent space. Moving a latent code along these vectors corresponds to high-level changes like urbanization growth, road reinforcement, or flora density.
  4. Data Augmentation: These directions are used to generate synthetic variations of rare classes to train more robust scene classifiers.

Datasets

The study utilizes three standard benchmarks for remote sensing, which the authors artificially imbalanced to test the effectiveness of their augmentation:

  • NWPU-RESISC45: 31,500 images across 45 classes.
  • AID (Aerial Image Dataset): 10,000 images across 30 classes.
  • UCMerced: 2,100 images across 21 classes.

The authors created specific "imbalanced" variants (e.g., Resisc10, where some classes have only 10 training samples) to simulate real-world scenarios where data for certain land-use types is scarce.

Key Insights and Ideas

  • Locality vs. Globality: LPP-derived directions are mathematically distinct from PCA directions (often separated by angles of 40° to 60°). This confirms that what is "locally important" is not the same as what is "globally variant."
  • Robustness to Artifacts: Qualitative results show that PCA-based edits often create "noisy" or "broken" textures in satellite images at high magnitudes. LPP-based edits maintain the structural integrity of the scene much better.
  • Generalization: LPP is a generalization of PCA. If you define the "neighborhood" as the entire dataset, LPP mathematically collapses into PCA. By narrowing the focus to local neighborhoods, the model captures more nuanced, class-preserving transformations.
  • Complementary Nature: The method works well alongside traditional geometric augmentations (flips/rotations), suggesting that "latent-space augmentation" provides a different kind of diversity that classifiers find useful.

Key Results

The LPP-based method consistently outperformed both baseline geometric augmentations and PCA-based methods in scene classification:

  • Accuracy Boost: On the highly imbalanced Resisc-10 dataset, LPP-based augmentation achieved an accuracy of 0.868, compared to 0.855 for PCA and 0.847 for the baseline.
  • Imbalanced Class Performance: The "Imbalanced Accuracy" (accuracy specifically on the rare classes) saw significant gains. For Resisc-10, LPP reached 0.611, while PCA only managed 0.587.
  • Class Preservation: In "Experiment V," the authors showed that LPP-augmented images were much more likely to be correctly classified as their original class by a pre-trained model, proving that LPP is better at "staying within the lines" of a specific category.

Limitations and Future Work

  • Saturation Point: The authors noted a "point of diminishing returns" where adding more context-based synthetic data no longer improves accuracy. This suggests there is a limit to how much information can be squeezed out of a single pre-trained GAN.
  • Linearity: LPP is a linear dimensionality reduction technique. While effective, the latent space of a GAN is non-linear; exploring non-linear manifold learning techniques (like UMAP or t-SNE variants) for latent steering remains an open area.
  • Human-in-the-loop: While the directions are "interpretable," they still require a human to look at the results and label the direction (e.g., "this one seems to be about roads"). Automating this semantic labeling is a logical next step.

Summary updated

Abstract

We present a locality-aware method for interpreting the latent space of wavelet-based generative adversarial networks (GANs), that can well capture the large spatial and spectral variability that is characteristic of satellite imagery. By focusing on preserving locality, the proposed method is able to decompose the weight-space of pre-trained GANs and recover interpretable directions that correspond to high-level semantic concepts (such as urbanization, structure density, flora presence)-that can subsequently be used for guided synthesis of satellite imagery. In contrast to typically used approaches that focus on capturing the variability of the weight-space in a reduced dimensionality space (i.e., based on principal component analysis (PCA)), we show that preserving locality leads to vectors with different angles, that are more robust to artifacts and can better preserve class information. Via a set of quantitative and qualitative examples, we further show that the proposed approach can outperform both baseline geometric augmentations, as well as global, PCA-based approaches for data synthesis in the context of data augmentation for satellite scene classification.
BibTeX
@article{kourmouli2024locality,
  author = {Georgia Kourmouli and Nikos Kostagiolas and Mihalis A. Nicolaou and Yannis Panagakis},
  title = {{Locality-Preserving Directions for Interpreting the Latent Space of Satellite Image GANs}},
  journal = {IEEE Geosci. Remote. Sens. Lett.},
  year = {2024},
  url = {https://openreview.net/forum?id=0YAeblZk7W},
}
Figure