Cross-View Image Synthesis with Deformable Convolution and Attention Mechanism

Hao Ding, Songsong Wu, Hao Tang, Fei Wu, Guangwei Gao, Xiao-Yuan Jing

Abstract

Cross-view image synthesis, the task of generating images from one perspective (e.g., aerial) given another (e.g., ground-level), faces significant challenges due to drastic geometric transformations and minimal visual overlap. This paper introduces an enhanced Generative Adversarial Network (GAN) that incorporates deformable convolutions and a dual-attention mechanism. These innovations allow the model to better handle view-invariant features and refine intermediate representations, resulting in sharper, more structurally accurate synthetic images.

Introduction

Translating images between aerial and ground-level views is a "daunting task" in computer vision. Objects viewed from above, such as building roofs, appear fundamentally different from their side-on facades seen at street level. This discrepancy, known as view invariance, often leads to blurry results and structural inaccuracies in synthesized images. While previous models like SelectionGAN utilized semantic maps for guidance, they frequently struggled with varying object scales and unclear boundaries.

Overview of the cross-view image synthesis task, showing transitions between aerial and ground-level perspectives.

Proposed Methodology

The authors propose a two-stage GAN architecture that builds upon the foundations of SelectionGAN while introducing three critical technical advancements:

1. Deformable Convolutions

Standard convolutional layers operate on a fixed grid, which is often insufficient for the complex geometric warping required in cross-view synthesis. By implementing deformable convolutions, the network learns offsets for its sampling positions. This flexibility allows the model to extract features more effectively from objects regardless of their shape or scale distortion during the perspective shift.

2. Dual-Attention Mechanism

To improve feature refinement between the two generation stages, the authors integrated an attention module comprising both Channel Attention and Spatial Attention. This module identifies the most relevant "what" and "where" features, refining the bottleneck representations before the final high-resolution image is generated.

The proposed two-stage architecture featuring the attention mechanism between Stage I and Stage II.

3. Semantic-Guided Discriminator

The model employs a second discriminator ($D_2$) that evaluates the generated image concatenated with its corresponding semantic map. This ensures that synthesized visual elements, such as trees or windows, align logically with the intended scene structure, reducing spatial artifacts.

Experimental Evaluation

The model was rigorously tested on the Dayton dataset, which contains 76,048 images. Experiments covered both aerial-to-ground (a2g) and ground-to-aerial (g2a) synthesis at resolutions of $64 \times 64$ and $256 \times 256$ pixels.

Key Results

The proposed method consistently outperformed baselines like Pix2Pix, X-Fork, and SelectionGAN:

  • Structural Similarity (SSIM): On the $256 \times 256$ a2g task, the model achieved an SSIM of 0.6524, significantly higher than SelectionGAN’s 0.5938.
  • Peak Signal-to-Noise Ratio (PSNR): The model reached 24.40 dB, indicating superior image fidelity.
  • Qualitative Quality: Visual inspections revealed sharper house boundaries, clearer road markings, and more realistic environmental details compared to previous blurry outputs.

Conclusion

By combining deformable convolutions with a dual-attention mechanism, this work successfully addresses the geometric and structural challenges of cross-view image synthesis. While the model relies on semantic maps and introduces additional computational complexity, the gains in image sharpness and structural integrity represent a significant step forward for view-invariant image generation. Future work may focus on reducing dependency on semantic maps and optimizing inference speed for real-time applications.

Summary updated

Abstract

Learning to generate natural scenes has always been a daunting task in computer vision. This is even more laborious when generating images with very different views. When the views are very different, the view fields have little overlap or objects are occluded, leading the task very challenging. In this paper, we propose to use Generative Adversarial Networks (GANs) based on a deformable convolution and attention mechanism to solve the problem of cross-view image synthesis (see Fig. 1). It is difficult to understand and transform scenes appearance and semantic information from another view, thus we use deformed convolution in the U-net network to improve the network’s ability to extract features of objects at different scales. Moreover, to better learn the correspondence between images from different views, we apply an attention mechanism to refine the intermediate feature map thus generating more realistic images. A large number of experiments on different size images on the Dayton dataset [1] show that our model can produce better results than state-of-the-art methods.
BibTeX
@inproceedings{ding2020cross,
  author = {Hao Ding and Songsong Wu and Hao Tang and Fei Wu and Guangwei Gao and Xiao-Yuan Jing},
  title = {{Cross-View Image Synthesis with Deformable Convolution and Attention Mechanism}},
  booktitle = {PRCV},
  year = {2020},
  url = {https://openreview.net/forum?id=zs-SSYTsFQO},
}
Figure