Abstract
Cross-view image synthesis, the task of generating images from one perspective (e.g., aerial) given another (e.g., ground-level), faces significant challenges due to drastic geometric transformations and minimal visual overlap. This paper introduces an enhanced Generative Adversarial Network (GAN) that incorporates deformable convolutions and a dual-attention mechanism. These innovations allow the model to better handle view-invariant features and refine intermediate representations, resulting in sharper, more structurally accurate synthetic images.
Introduction
Translating images between aerial and ground-level views is a "daunting task" in computer vision. Objects viewed from above, such as building roofs, appear fundamentally different from their side-on facades seen at street level. This discrepancy, known as view invariance, often leads to blurry results and structural inaccuracies in synthesized images. While previous models like SelectionGAN utilized semantic maps for guidance, they frequently struggled with varying object scales and unclear boundaries.
Proposed Methodology
The authors propose a two-stage GAN architecture that builds upon the foundations of SelectionGAN while introducing three critical technical advancements:
1. Deformable Convolutions
Standard convolutional layers operate on a fixed grid, which is often insufficient for the complex geometric warping required in cross-view synthesis. By implementing deformable convolutions, the network learns offsets for its sampling positions. This flexibility allows the model to extract features more effectively from objects regardless of their shape or scale distortion during the perspective shift.
2. Dual-Attention Mechanism
To improve feature refinement between the two generation stages, the authors integrated an attention module comprising both Channel Attention and Spatial Attention. This module identifies the most relevant "what" and "where" features, refining the bottleneck representations before the final high-resolution image is generated.
3. Semantic-Guided Discriminator
The model employs a second discriminator ($D_2$) that evaluates the generated image concatenated with its corresponding semantic map. This ensures that synthesized visual elements, such as trees or windows, align logically with the intended scene structure, reducing spatial artifacts.
Experimental Evaluation
The model was rigorously tested on the Dayton dataset, which contains 76,048 images. Experiments covered both aerial-to-ground (a2g) and ground-to-aerial (g2a) synthesis at resolutions of $64 \times 64$ and $256 \times 256$ pixels.
Key Results
The proposed method consistently outperformed baselines like Pix2Pix, X-Fork, and SelectionGAN:
- Structural Similarity (SSIM): On the $256 \times 256$ a2g task, the model achieved an SSIM of 0.6524, significantly higher than SelectionGAN’s 0.5938.
- Peak Signal-to-Noise Ratio (PSNR): The model reached 24.40 dB, indicating superior image fidelity.
- Qualitative Quality: Visual inspections revealed sharper house boundaries, clearer road markings, and more realistic environmental details compared to previous blurry outputs.
Conclusion
By combining deformable convolutions with a dual-attention mechanism, this work successfully addresses the geometric and structural challenges of cross-view image synthesis. While the model relies on semantic maps and introduces additional computational complexity, the gains in image sharpness and structural integrity represent a significant step forward for view-invariant image generation. Future work may focus on reducing dependency on semantic maps and optimizing inference speed for real-time applications.
Summary updated