Exocentric to Egocentric Image Generation Via Parallel Generative Adversarial Network

Gaowen Liu, Hao Tang, Hugo Latapie, Yan Yan

Abstract

Generating first-person (egocentric) images from third-person (exocentric) views is a significant challenge in computer vision due to drastic perspective differences and minimal visual overlap. This paper presents a Parallel Generative Adversarial Network (P-GAN) designed to bridge this gap. By employing hard-shared network layers, a novel cross-cycle loss, and a contextual feature loss, P-GAN effectively aligns latent representations across disparate viewpoints. Experimental results on the Exo-Ego dataset demonstrate that P-GAN significantly outperforms existing cross-view generation methods in both structural coherence and semantic accuracy.

Introduction

The rise of wearable cameras has made understanding the relationship between exocentric (third-person) and egocentric (first-person) viewpoints a highly relevant computer vision challenge. Traditional image-to-image translation models, such as those converting day to night, typically rely on shared underlying geometry. However, translating an exocentric view to an egocentric one is an "extreme" cross-view task. These perspectives share very little spatial overlap; an observer's view of a person is fundamentally different from that person's own visual field. Existing cross-view GANs often struggle with this disparity, frequently producing blurry or structurally incoherent images.

Methodology

To overcome the lack of spatial overlap, we introduce the Parallel Generative Adversarial Network (P-GAN). The architecture utilizes two parallel U-Net generators and two PatchGAN discriminators.

The P-GAN architecture, highlighting parallel generators with hard-shared encoder layers and cross-cycle loss for latent alignment.

Key Innovations

  • Hard-Sharing of Layers: The first three layers of the down-sampling encoders for both U-Net generators share identical structures and weights. This forces the networks to learn a common, viewpoint-invariant high-level semantic representation of the scene.
  • Cross-Cycle Loss: P-GAN leverages the U-Net structure to ensure latent alignment. By passing an image through the encoder of View A and decoding it with the decoder of View B, the network is penalized if it cannot reconstruct the original image, guaranteeing that the parallel networks share a consistent latent space.
  • Contextual Feature Loss: Since spatial coordinates do not align between views, standard pixel-wise L1 loss is insufficient. We employ a contextual loss that compares high-level feature maps from a pre-trained VGG-19 network, measuring semantic and textural similarity regardless of spatial position.

Experimental Setup

The model is trained and evaluated on the Exo-Ego dataset, which contains paired images captured simultaneously from exocentric and wearable cameras.

  • Side2Ego Subset: Side-view to egocentric view (26,764 training pairs; 13,788 testing pairs).
  • Top2Ego Subset: Top-down view to egocentric view (28,408 training pairs; 14,064 testing pairs). Images were scaled to 256×256 pixels for training.

Results and Discussion

P-GAN was compared against single-view models (Pix2Pix, CycleGAN) and state-of-the-art cross-view models (X-Fork, X-Seq, SelectionGAN).

Quantitative Performance

P-GAN consistently achieved the best performance across multiple metrics. On the Top2Ego dataset, P-GAN achieved an SSIM of 0.5287 and a Top-1 Accuracy of 9.76%. On the Side2Ego dataset, it reached an SSIM of 0.5205 and a Top-1 Accuracy of 20.96%.

Qualitative Analysis

Visually, P-GAN places objects in the correct egocentric positions much more reliably than baseline models. While competitors often output blurry blobs, P-GAN's outputs maintain recognizable structures and textures that better match the ground truth.

Qualitative results comparing P-GAN with baseline methods on indoor scenes. P-GAN demonstrates superior structural and textural fidelity.

Key Insights

  • Forced Latent Alignment: Hard-sharing early encoder weights is crucial for stripping away viewpoint-specific geometry and focusing on shared scene semantics.
  • Internal Cycle Consistency: The cross-cycle loss ensures that both parallel networks are effectively "speaking the same language" in their latent representations.
  • Feature-Space Optimization: Utilizing contextual loss in the VGG-19 feature space allows the model to capture the scene's essence without being penalized for the inherent spatial misalignment of the viewpoints.

Conclusion

P-GAN represents a significant step forward in extreme cross-view image generation. While generating perfectly photorealistic egocentric images remains an open challenge, P-GAN's architectural innovations provide a robust framework for aligning disparate perspectives. Future work could involve applying this parallel architecture to video generation for temporal consistency or testing the mechanism on other extreme cross-view tasks like ground-to-satellite imagery.

Summary updated

BibTeX
@inproceedings{liu2020exocentric,
  author = {Gaowen Liu and Hao Tang and Hugo Latapie and Yan Yan},
  title = {{Exocentric to Egocentric Image Generation Via Parallel Generative Adversarial Network}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9053957},
}
Figure