Cascaded Cross MLP-Mixer GANs for Cross-View Image Translation

Bin Ren, Hao Tang, Nicu Sebe

Introduction

Cross-view image translation involves generating a ground-level street view from an overhead aerial image, or vice-versa. This task is significantly more complex than standard image-to-image translation because it requires modeling extreme geometric transformations and drastic appearance changes. Traditional approaches using Convolutional Neural Networks (CNNs) often fail to capture the long-range spatial dependencies necessary for such large-scale pixel displacement. While Transformer-based models offer a solution through self-attention, their quadratic computational complexity makes them difficult to scale for high-resolution generation.

Proposed Framework: CrossMLP-GAN

To address these challenges, this paper introduces a two-stage Generative Adversarial Network (GAN) framework centered around Cascaded Cross MLP-Mixer (CrossMLP) blocks. This architecture provides an efficient alternative to both CNNs and Transformers by utilizing Multi-Layer Perceptrons (MLPs) to model spatial and channel-wise dependencies.

Overview of the two-stage Cascaded CrossMLP framework showing Stage I and Stage II.

Stage I: Cascaded CrossMLP Sub-network

The first stage employs a sequence of CrossMLP blocks to iteratively refine the translation. Each block performs two primary operations:

  1. Token-mixing: Captures long-range spatial dependencies by mixing information across different image patches.
  2. Channel-mixing: Integrates information across feature dimensions at each spatial location.

A key innovation is the Cross-Modal Interaction within these blocks. The model simultaneously processes the source image and its corresponding semantic map, learning latent translation cues that guide the placement of features in the target view. This cascading approach mimics a progressive sketching process, allowing the geometry to stabilize before final rendering.

Stage II: Refinement and Robust Loss

The second stage refines the coarse output from Stage I. To handle the "noisy" or inaccurate semantic labels often found in real-world datasets, the authors introduce a refined pixel-level loss. This loss function incorporates a consistency constraint between the two stages, making the optimization process more robust against label errors and leading to sharper final images.

Experimental Results

The model was evaluated on the Dayton and CVUSA datasets. The Dayton dataset consists of over 76,000 aerial-ground pairs, while CVUSA provides panoramic ground-level views.

Quantitative comparison on the Dayton dataset demonstrating state-of-the-art performance.

As shown in the results for the Dayton dataset, the proposed CrossMLP method significantly outperforms existing state-of-the-art models such as Pix2pix, X-Fork, and SelectionGAN. Notably, it achieved a Top-1 accuracy of 47.65% and an Inception Score (IS) of 3.3466, moving closer to the performance of real data than previous methods. Qualitative analysis confirms that CrossMLP produces images with fewer artifacts and more realistic textures in complex regions like foliage and building facades.

Conclusion

By replacing traditional attention mechanisms with efficient MLP-Mixer blocks and employing a cascaded refinement strategy, the CrossMLP-GAN sets a new benchmark for cross-view image translation. The framework successfully balances computational efficiency with the ability to model the complex geometric shifts inherent in geospatial view synthesis.

Summary updated

BibTeX
@inproceedings{ren2021cascaded,
  author = {Bin Ren and Hao Tang and Nicu Sebe},
  title = {{Cascaded Cross MLP-Mixer GANs for Cross-View Image Translation}},
  booktitle = {BMVC},
  year = {2021},
  url = {https://openreview.net/forum?id=AuDkKlQ2zYdS},
}
Figure