Txt2Img-MHN: Remote Sensing Image Generation from Text Using Modern Hopfield Networks

Yonghao Xu, Weikang Yu, Pedram Ghamisi, Michael Kopp, Sepp Hochreiter

Abstract

Generating high-fidelity remote sensing (RS) images from text descriptions is a significant challenge due to complex spatial distributions and the inherent modality gap between abstract text and high-dimensional pixel data. This paper introduces Txt2Img-MHN, a novel framework that utilizes Modern Hopfield Networks (MHNs) to bridge this gap through hierarchical prototype learning. By representing semantic concepts as retrievable patterns, the model captures complex spatial relationships and object counts more effectively than traditional generative models. Evaluated on the RSICD dataset, Txt2Img-MHN achieves a state-of-the-art zero-shot classification accuracy of 65.72%, demonstrating superior semantic consistency over existing GAN and Transformer-based approaches.

Introduction

Remote sensing (RS) image generation from text prompts is more complex than natural image synthesis. RS scenes often contain intricate spatial arrangements, varying resolutions, and specific semantic requirements—such as distinguishing between "a few" and "many" buildings. Traditional Generative Adversarial Networks (GANs) frequently fail to maintain high-level semantic consistency, often producing visually appealing but semantically inaccurate results. Furthermore, standard metrics like Inception Score (IS) and Fréchet Inception Distance (FID) prioritize visual quality over semantic alignment, necessitating new evaluation paradigms that measure how well an image matches its textual description.

Methodology

The core of Txt2Img-MHN is the integration of Modern Hopfield Networks (MHNs) within a hierarchical prototype learning framework. Unlike classical binary Hopfield networks, MHNs are continuous and differentiable, allowing for the storage and retrieval of exponentially many patterns in a single update step.

Hierarchical Prototype Learning

The model maps text descriptions to "prototypes"—representative anchors in a joint text-image embedding space. This coarse-to-fine strategy allows the model to first identify the general scene type before synthesizing fine-grained details. The architecture features a Prototype Learning Block, which combines a Hopfield Layer for pattern lookup with a Self-Attention Layer to enhance internal token interactions.

Tokenization and Generation

Txt2Img-MHN employs Vector Quantized Variational Autoencoders (VQVAE) or VQGANs to discretize images into tokens. The MHN then predicts these image tokens autoregressively based on text input. While VQVAE excels at preserving basic geometric shapes, VQGAN provides superior textures and spatial resolution through adversarial training, though it occasionally risks "hallucinating" shapes, such as rendering circular storage tanks as rectangles.

Results and Discussion

The framework was evaluated using the RSICD (Remote Sensing Image Captioning Dataset), which contains over 10,000 aerial images across 30 scene classes.

Semantic Consistency and Visual Fidelity

To rigorously assess semantic alignment, the authors proposed a Zero-Shot Classification Overall Accuracy (OA) metric. A classifier (ResNet-18) was trained solely on synthetic images and tested on real data; Txt2Img-MHN achieved an OA of 65.72%, significantly outperforming DALL-E (28.59%) and the previous state-of-the-art DF-GAN (51.99%).

Examples of remote sensing images generated by Txt2Img-MHN across various scene categories, demonstrating strong semantic alignment with text prompts.

Qualitative analysis reveals that Txt2Img-MHN accurately reflects quantitative descriptors in prompts. For example, it can distinguish between "some" and "many" objects, a task where competing models often struggle. As shown in the generated samples, the model handles diverse environments ranging from dense residential areas to industrial storage sites with high fidelity.

Ablation Study

Ablation experiments confirmed the necessity of the model's components. Removing the Hopfield layer resulted in a drop in OA to 57.10%, while the removal of the self-attention layer caused a drastic decline to 36.50%, highlighting the vital synergy between pattern retrieval and token interaction.

Conclusion

Txt2Img-MHN represents a significant advancement in text-to-image synthesis for remote sensing. By leveraging the associative memory capabilities of Modern Hopfield Networks, the model effectively bridges the modality gap and achieves high semantic consistency. Future work will focus on utilizing unlabeled high-resolution imagery to further enhance generative performance and addressing geometric distortions in VQGAN-based synthesis.

Summary updated

BibTeX
@article{xu2023txt,
  author = {Yonghao Xu and Weikang Yu and Pedram Ghamisi and Michael Kopp and Sepp Hochreiter},
  title = {{Txt2Img-MHN: Remote Sensing Image Generation from Text Using Modern Hopfield Networks}},
  journal = {IEEE Trans. Image Process.},
  year = {2023},
  url = {https://openreview.net/forum?id=TZ_4lzlxZ1K},
}
Figure