PI-Trans: Parallel-ConvMLP and Implicit-Transformation Based GAN for Cross-View Image Translation

Bin Ren, Hao Tang, Yiming Wang, Xia Li, Wei Wang, Nicu Sebe

Abstract

For semantic-guided cross-view image translation, it is crucial to learn where to sample pixels from the source view image and where to reallocate them guided by the target view semantic map, especially when there is little overlap or drastic view difference between the source and target images. Hence, one not only needs to encode the long-range dependencies among pixels in both the source view image and target view semantic map but also needs to translate these learned dependencies. To this end, we propose a novel generative adversarial network, PI-Trans, which mainly consists of a novel Parallel-ConvMLP module and an Implicit Transformation module at multiple semantic levels. Extensive experimental results show that PI-Trans achieves the best qualitative and quantitative performance by a large margin compared to the state-of-the-art methods on two challenging datasets.

Introduction

Cross-view image translation—specifically generating a street-level view from a satellite or aerial image—is a notoriously difficult task in computer vision. Unlike standard image-to-image translation, cross-view tasks involve drastic geometric shifts and very little pixel-to-pixel overlap. Traditional Convolutional Neural Networks (CNNs) often fall into a "locality trap," where small kernels struggle to model the long-range dependencies required to map an aerial roof to a ground-level facade. Conversely, while Multi-Layer Perceptrons (MLPs) can capture global context, they often sacrifice the fine-grained spatial details necessary for realistic synthesis.

PI-Trans addresses these challenges by combining convolutional layers with a unique parallel MLP structure and an implicit transformation module, effectively mapping long-range pixel dependencies while preserving sharp spatial details.

Methodology

PI-Trans utilizes a dual-branch architecture that integrates semantic guidance with raw image translation. The framework is built upon two primary innovations:

1. Parallel-ConvMLP Module

Instead of stacking convolutions and MLPs sequentially, PI-Trans uses a parallel approach. The input feature map is split into two chunks: a Channel-wise MLP that focuses on identifying objects ("what") and a Spatial-wise MLP that focuses on their positioning ("where"). By processing these in parallel alongside convolutional layers, the model maintains sharp spatial boundaries while reasoning about the global layout of the scene.

2. Implicit Transformation Module

This module acts as a sophisticated fusion center, taking inputs from the target semantic map, the aerial image features, and a "direct translation" branch. While the direct translation branch (image-to-image without semantic guidance) may produce structurally incorrect results, it serves as a vital "color bank." The Implicit Transformation module uses an attention-based mechanism where the semantic map selects the most relevant color and texture information from this branch to populate the final ground-level image.

The PI-Trans architecture featuring the Direct Translation Branch and Implicit Transformation modules.

Experimental Results

The model was evaluated on two standard benchmarks: Dayton (urban aerial/ground pairs) and CVUSA (large-scale satellite/panorama pairs). PI-Trans achieved new state-of-the-art (SOTA) results across both datasets.

Quantitative Performance

On the Dayton dataset, PI-Trans achieved a Top-1 accuracy of 49.48%, significantly outperforming previous models like CrossMLP. It also showed marked improvements in Inception Score (IS) and LPIPS, indicating that the generated images are both more realistic and perceptually similar to ground truth data.

Qualitative Superiority

Visual comparisons demonstrate that PI-Trans avoids common artifacts such as "ghosting" or structural distortions that plague earlier models. By effectively leveraging the semantic map and the direct translation color bank, the model produces sharper images with more accurate global layouts.

Quantitative comparison and qualitative results on the Dayton and CVUSA datasets.

Conclusion and Future Work

PI-Trans demonstrates that decoupling spatial and channel reasoning into parallel paths prevents the "vanishing detail" problem common in deep MLP architectures. This allows the model to capture global context without losing local sharpness. While the model relies on accurate target semantic maps, its success suggests that the parallel MLP-Conv approach could be extended to other difficult synthesis tasks, such as 3D-aware image synthesis or indoor scene generation.

Summary updated

Abstract

For semantic-guided cross-view image translation, it is crucial to learn where to sample pixels from the source view image and where to reallocate them guided by the target view semantic map, especially when there is little overlap or drastic view difference between the source and target images. Hence, one not only needs to encode the long-range dependencies among pixels in both the source view image and target view semantic map but also needs to translate these learned dependencies. To this end, we propose a novel generative adversarial network, PI-Trans, which mainly consists of a novel Parallel-ConvMLP module and an Implicit Transformation module at multiple semantic levels. Extensive experimental results show that PI-Trans achieves the best qualitative and quantitative performance by a large margin compared to the state-of-the-art methods on two challenging datasets. The source code is available at https://github.com/Amazingren/PI-Trans.
BibTeX
@inproceedings{ren2023pi,
  author = {Bin Ren and Hao Tang and Yiming Wang and Xia Li and Wei Wang and Nicu Sebe},
  title = {{PI-Trans: Parallel-ConvMLP and Implicit-Transformation Based GAN for Cross-View Image Translation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095449},
}
Figure