Abstract
Significant advancements have been made in semantic image synthesis in remote sensing. However, existing methods still face formidable challenges in balancing semantic controllability and diversity. In this paper, we present a Hybrid Semantic Embedding Guided Generative Adversarial Network (HySEGGAN) for controllable and efficient remote sensing image synthesis. Specifically, HySEGGAN leverages hierarchical information from a single source. Motivated by feature description, we propose a hybrid semantic Embedding method, that coordinates fine-grained local semantic layouts to characterize the geometric structure of remote sensing objects without extra information. Besides, a Semantic Refinement Network (SRN) is introduced, incorporating a novel loss function to ensure fine-grained semantic feedback. The proposed approach mitigates semantic confusion and prevents geometric pattern collapse. Experimental results indicate that the method strikes an excellent balance between semantic controllability and diversity. Furthermore, HySEGGAN significantly improves the quality of synthesized images and achieves state-of-the-art performance as a data augmentation technique across multiple datasets for downstream tasks.
Introduction
Generating realistic satellite imagery from semantic masks is a vital task for urban planning and data augmentation. However, remote sensing data presents unique challenges that standard generative adversarial networks (GANs) often fail to address. First, semantic ambiguity arises because classes like "grass" and "forest" share similar spectral features. Second, geometric complexity in man-made structures like buildings leads to "pattern collapse" or distortions. Finally, there is a persistent trade-off between controllability and diversity: forcing strict adherence to a mask often results in repetitive, unrealistic textures, while prioritizing diversity can lead the model to ignore the input labels.
Methodology: The HySEGGAN Framework
The Hybrid Semantic Embedding Guided GAN (HySEGGAN) addresses these issues by moving beyond flat semantic labels. It treats geometry as a primary feature, extracted directly from the input mask without requiring auxiliary data like sketches.
Geometric-informed Spatial Descriptor (GSD)
Instead of simple one-hot encoding, HySEGGAN calculates a "shape fingerprint" for every pixel. By using polar coordinates relative to object contours, the GSD creates a histogram describing local geometry. This allows the generator to understand its position within an object (e.g., distinguishing a corner from a center), leading to sharper edges and more realistic internal textures without needing extra inputs.
Hybrid Semantic Feature Modulation (HSFM)
The HSFM block integrates GSD with traditional semantic embeddings. It employs "Semantics Decoupling" to scale these features before they enter adaptive normalization layers. This ensures the model "paints" textures that respect both the semantic class and the underlying local shape.
Semantic Refinement Network (SRN)
Standard discriminators focus on global realism. The SRN, however, provides pixel-level feedback by asking if each generated pixel matches its intended label. This fine-grained supervision is critical for maintaining alignment in complex, multi-class remote sensing scenes.
Experimental Evaluation
The model was evaluated on two major benchmarks: GID-15 (Gaofen-2 satellite data) and ISPRS Potsdam (high-resolution aerial data).
Key Results
- Image Quality: On GID-15, HySEGGAN achieved a Fréchet Inception Distance (FID) of 132.28, outperforming established models like SPADE (142.57) and INADE (136.61).
- Controllability: The model reached a mean Intersection over Union (mIoU) of 71.23%, a significant improvement over the previous state-of-the-art (~64%), demonstrating superior alignment between synthetic images and input masks.
- Downstream Utility: Using HySEGGAN-generated images for data augmentation improved downstream segmentation performance by +2.32% mIoU on GID-15, proving its value as a "force multiplier" for training robust models.
Conclusion
HySEGGAN successfully reconciles the tension between semantic controllability and generative diversity. By explicitly encoding geometric structure via the GSD and providing fine-grained feedback through the SRN, it produces high-quality, diverse remote sensing imagery. Future work will address "dominance confusion" in scenes where a single background class overwhelms the layout, further refining the mathematical description of shapes to handle such cases.
Summary updated