Abstract
Generating first-person (egocentric) images from third-person (exocentric) views is a significant challenge in computer vision due to drastic perspective differences and minimal visual overlap. This paper presents a Parallel Generative Adversarial Network (P-GAN) designed to bridge this gap. By employing hard-shared network layers, a novel cross-cycle loss, and a contextual feature loss, P-GAN effectively aligns latent representations across disparate viewpoints. Experimental results on the Exo-Ego dataset demonstrate that P-GAN significantly outperforms existing cross-view generation methods in both structural coherence and semantic accuracy.
Introduction
The rise of wearable cameras has made understanding the relationship between exocentric (third-person) and egocentric (first-person) viewpoints a highly relevant computer vision challenge. Traditional image-to-image translation models, such as those converting day to night, typically rely on shared underlying geometry. However, translating an exocentric view to an egocentric one is an "extreme" cross-view task. These perspectives share very little spatial overlap; an observer's view of a person is fundamentally different from that person's own visual field. Existing cross-view GANs often struggle with this disparity, frequently producing blurry or structurally incoherent images.
Methodology
To overcome the lack of spatial overlap, we introduce the Parallel Generative Adversarial Network (P-GAN). The architecture utilizes two parallel U-Net generators and two PatchGAN discriminators.
Key Innovations
- Hard-Sharing of Layers: The first three layers of the down-sampling encoders for both U-Net generators share identical structures and weights. This forces the networks to learn a common, viewpoint-invariant high-level semantic representation of the scene.
- Cross-Cycle Loss: P-GAN leverages the U-Net structure to ensure latent alignment. By passing an image through the encoder of View A and decoding it with the decoder of View B, the network is penalized if it cannot reconstruct the original image, guaranteeing that the parallel networks share a consistent latent space.
- Contextual Feature Loss: Since spatial coordinates do not align between views, standard pixel-wise L1 loss is insufficient. We employ a contextual loss that compares high-level feature maps from a pre-trained VGG-19 network, measuring semantic and textural similarity regardless of spatial position.
Experimental Setup
The model is trained and evaluated on the Exo-Ego dataset, which contains paired images captured simultaneously from exocentric and wearable cameras.
- Side2Ego Subset: Side-view to egocentric view (26,764 training pairs; 13,788 testing pairs).
- Top2Ego Subset: Top-down view to egocentric view (28,408 training pairs; 14,064 testing pairs). Images were scaled to 256×256 pixels for training.
Results and Discussion
P-GAN was compared against single-view models (Pix2Pix, CycleGAN) and state-of-the-art cross-view models (X-Fork, X-Seq, SelectionGAN).
Quantitative Performance
P-GAN consistently achieved the best performance across multiple metrics. On the Top2Ego dataset, P-GAN achieved an SSIM of 0.5287 and a Top-1 Accuracy of 9.76%. On the Side2Ego dataset, it reached an SSIM of 0.5205 and a Top-1 Accuracy of 20.96%.
Qualitative Analysis
Visually, P-GAN places objects in the correct egocentric positions much more reliably than baseline models. While competitors often output blurry blobs, P-GAN's outputs maintain recognizable structures and textures that better match the ground truth.
Key Insights
- Forced Latent Alignment: Hard-sharing early encoder weights is crucial for stripping away viewpoint-specific geometry and focusing on shared scene semantics.
- Internal Cycle Consistency: The cross-cycle loss ensures that both parallel networks are effectively "speaking the same language" in their latent representations.
- Feature-Space Optimization: Utilizing contextual loss in the VGG-19 feature space allows the model to capture the scene's essence without being penalized for the inherent spatial misalignment of the viewpoints.
Conclusion
P-GAN represents a significant step forward in extreme cross-view image generation. While generating perfectly photorealistic egocentric images remains an open challenge, P-GAN's architectural innovations provide a robust framework for aligning disparate perspectives. Future work could involve applying this parallel architecture to video generation for temporal consistency or testing the mechanism on other extreme cross-view tasks like ground-to-satellite imagery.
Summary updated