Aerial Diffusion: Text Guided Ground-to-Aerial View Translation from a Single Image using Diffusion Models

Divya Kothandaraman, Tianyi Zhou, Ming C. Lin, Dinesh ManochaarXiv 2023 · DOI · OpenReview

TL;DR

This paper introduces "Aerial Diffusion," a novel method for generating aerial views from a single ground-view image using text guidance. It tackles the significant domain gap between ground and aerial perspectives by employing a homography transformation and a unique alternating sampling strategy within a diffusion model, achieving unsupervised ground-to-aerial translation.

What Problem Does This Paper Address?

The core problem this paper addresses is the challenging task of transforming a single ground-view image into a corresponding aerial-view image. This is crucial for applications like urban planning, surveillance, and content creation, where aerial imagery is valuable but often scarce or difficult to obtain. Existing methods typically require paired ground-aerial data, multiple views, or depth information, which are not always available. Furthermore, the significant difference in perspective and domain between ground and aerial views creates a substantial "domain gap" that makes direct translation difficult for existing text-to-image models. This gap leads to models either producing images that are not truly aerial or losing fidelity to the original ground-view details.

Methods

Aerial Diffusion leverages a pretrained text-to-image diffusion model as its backbone. The method introduces several key innovations to overcome the domain gap and achieve unsupervised ground-to-aerial translation:

  • Homography Preprocessing: Before finetuning, the ground-view image is transformed using an "inverse perspective mapping" homography. This creates a pseudo-aerial view that helps reduce the bias of the diffusion model towards the original ground-view perspective during finetuning.
  • Source Text Finetuning: Instead of using the target "aerial view" text for finetuning, the model uses the original "ground-view" text. This encourages the model to learn a text embedding that is closer to the image space of the ground-view, further mitigating bias and allowing for a more effective search for the aerial view.
  • Alternating Sampling Strategy: During inference, a novel alternating sampling strategy is employed. This strategy switches between two text embeddings: one that strongly enforces the "aerial view" perspective and another that balances fidelity to the original ground-view image. This alternating process allows the model to navigate the complex, high-dimensional manifold of aerial view generation more effectively than simple linear interpolation, leading to high-fidelity aerial images.

Datasets

The paper does not introduce new datasets. It utilizes existing images from various domains (nature, human actions, indoor scenes, etc.) for demonstration and evaluation. The core model is based on a pretrained diffusion model trained on large-scale text-image datasets like LAION-5B.

Key Insights and Ideas

  • Homography as a Bias Reducer: The homography transformation, while not producing a perfect aerial view, effectively shifts the image representation closer to the target domain, significantly reducing the model's inherent bias towards the ground-view.
  • Finetuning with Source Text is Crucial: Using the original ground-view text for finetuning, rather than the target aerial-view text, is key to preventing the model from developing a strong bias towards an incorrect aerial representation and helps it learn a more adaptable embedding.
  • Alternating Sampling for Complex Transformations: Simple linear interpolation in the text embedding space fails for large domain shifts like ground-to-aerial. The alternating sampling strategy is essential for navigating the non-linear manifold and achieving a good trade-off between viewpoint and fidelity.
  • Unsupervised Ground-to-Aerial is Achievable: The paper demonstrates that high-quality ground-to-aerial translation can be achieved without paired ground-aerial data, relying instead on the power of pretrained diffusion models and clever manipulation of text embeddings.

Key Results

  • Qualitative Success: Aerial Diffusion generates high-quality, high-fidelity aerial images from diverse ground-view inputs, preserving details from the original image while adopting the aerial perspective. The results are visually impressive across various categories, including animals, scenes, and human actions.
  • Quantitative Evaluation (User Study):
    • Image Quality: Participants rated the generated aerial-view images as high-fidelity with an average score of 3.289 on a 5-point Likert scale.
  • Alternating Sampling: 83.05% of participants preferred images generated by Aerial Diffusion over those from a baseline without alternating sampling.
  • Reference for Aerial Diffusion: 78.125% of participants preferred Aerial Diffusion's output over a variant that finetuned with the target aerial-view text.
  • Comparison to SOTA: The method significantly outperforms existing text-based image translation methods like IMAGIC, which struggle to generate true aerial views and often reproduce the ground-view with minor changes.

Limitations and Future Work

  • Limitations:
    • Directional Bias: The homography transformation can introduce a directional bias in the generated aerial images.
  • Domain Knowledge: The model is limited by the knowledge present in the pretrained diffusion model and cannot hallucinate scenes or objects outside of its training domain.
  • Hyperparameter Tuning: The parameter 'a' in the alternating sampling strategy requires manual tuning to balance fidelity and viewpoint.
  • Quantitative Metrics: The lack of standard benchmarks and quantitative metrics for ground-to-aerial translation makes objective evaluation challenging.
  • Future Work:
    • Develop methods to overcome the directional bias introduced by homography.
  • Extend the approach to handle complex scenes with multiple objects and intricate backgrounds more robustly.
  • Explore automatic hyperparameter tuning for the sampling strategy.
  • Investigate extensions to video, complex scene generation, and synthetic data generation for aerial video analysis.

Summary updated

Abstract

We present a novel method, Aerial Diffusion, for generating aerial views from a single ground-view image using text guidance. Aerial Diffusion leverages a pretrained text-image diffusion model for prior knowledge. We address two main challenges corresponding to domain gap between the ground-view and the aerial view and the two views being far apart in the text-image embedding manifold. Our approach uses a homography inspired by inverse perspective mapping prior to finetuning the pretrained diffusion model. Additionally, using the text corresponding to the ground-view to finetune the model helps us capture the details in the ground-view image at a relatively low bias towards the ground-view image. Aerial Diffusion uses an alternating sampling strategy to compute the optimal solution on complex high-dimensional manifold and generate a high-fidelity (w.r.t. ground view) aerial image. We demonstrate the quality and versatility of Aerial Diffusion on a plethora of images from various domains including nature, human actions, indoor scenes, etc. We qualitatively prove the effectiveness of our method with extensive ablations and comparisons. To the best of our knowledge, Aerial Diffusion is the first approach that performs ground-to-aerial translation in an unsupervised manner.
BibTeX
@misc{kothandaraman2023aerial,
  author = {Divya Kothandaraman and Tianyi Zhou and Ming C. Lin and Dinesh Manocha},
  title = {{Aerial Diffusion: Text Guided Ground-to-Aerial View Translation from a Single Image using Diffusion Models}},
  year = {2023},
  eprint = {2303.11444},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.11444},
}
Figure