AerialBooth: Mutual Information Guidance for Text Controlled Aerial View Synthesis from a Single Image

Divya Kothandaraman, Tianyi Zhou, Ming C. Lin, Dinesh Manocha

Introduction

Generating high-fidelity aerial views from a single ground-level photograph is a formidable challenge in computer vision. While modern text-to-image models can produce generic aerial scenes, they often struggle to preserve the specific semantic details of a provided subject, such as a particular lighthouse or a specific group of friends. Existing methods typically rely on extensive 3D data or struggle with the drastic perspective shifts required for aerial synthesis. HawkI (AerialBooth) introduces a "3D-free" framework that bridges this gap by combining classical geometric techniques with information theory.

Methodology: Bridging Geometry and Information Theory

HawkI utilizes a two-stage approach integrated with a pretrained Stable Diffusion model to achieve realistic perspective transformations without the need for massive 3D datasets.

1. Test-Time Optimization with Inverse Perspective Mapping (IPM)

The framework begins with a brief finetuning step on the input image. To guide the model toward an aerial perspective, the authors incorporate Inverse Perspective Mapping (IPM). Although IPM often produces distorted "top-down" views, these serve as a crucial geometric hint or "compass," breaking the model's inherent bias toward the original ground-level viewpoint.

2. Mutual Information (MI) Guided Inference

During the denoising process, HawkI employs a novel guidance strategy based on Mutual Information. Unlike standard pixel-wise metrics (like L2 distance), MI measures the statistical dependence between image distributions. This allows the model to enforce semantic consistency—ensuring the generated aerial view contains the same content as the input—without being constrained by the original pixel-level layout.

Key Results and Evaluation

The researchers evaluated the framework using two newly curated datasets: HawkI-Syn (500 synthetic images with LLM-generated prompts) and HawkI-Real (139 real-world photos from Unsplash). These datasets span diverse categories, including architecture, natural landscapes, and human actions.

Comparison of input ground-level images and the aerial views generated by HawkI across various subjects.

HawkI demonstrates significant qualitative and quantitative improvements over baseline methods like DreamBooth and Aerial Diffusion. It successfully generates images that are both "more aerial" in perspective and more faithful to the original subject's identity. Quantitatively, the method achieves a superior balance between text alignment (CLIP-Score) and image fidelity (DINO/SSCD). Notably, HawkI performs comparably to state-of-the-art 3D-trained models like Zero123++, despite requiring no 3D training data. Furthermore, using HawkI-generated data improved zero-shot action recognition for drones (UAVs) by 7.14%, proving the synthetic images' realism for machine learning applications.

Conclusion and Future Outlook

HawkI proves that the 3D knowledge embedded within 2D generative models can be effectively unlocked for novel view synthesis through classical computer vision "nudges." While the method currently faces limitations regarding hallucination and the need for more precise quantitative camera controls, its success in generating high-fidelity aerial views from single images marks a significant step forward for data augmentation and creative tools.

Summary updated

Abstract

We present a novel method, AerialBooth, for synthesizing the aerial view from a single input image using its text description. We leverage the pretrained text-to-2D image stable diffusion model as prior knowledge of the 3D world. The model is finetuned in two steps to optimize for the text embedding and the UNet that reconstruct the input image and its inverse perspective mapping respectively. The inverse perspective mapping creates variance within the text-image space of the diffusion model, while providing weak guidance for aerial view synthesis. At inference, we steer the contents of the generated image towards the input image using novel mutual information guidance that maximizes the information content between the probability distributions of the two images. We evaluate our approach on a wide spectrum of real and synthetic data, including natural scenes, indoor scenes, human action, etc. Through extensive experiments and ablation studies, we demonstrate the effectiveness of AerialBooth and also its generalizability to other text-controlled views. We also show that AerialBooth achieves the best viewpoint-fidelity trade-off though quantitative evaluation on 7 metrics analyzing viewpoint and fidelity w.r.t. input image. Code and data is available at https://github.com/divyakraman/AerialBooth2023.
BibTeX
@misc{kothandaraman2023aerialbooth,
  author = {Divya Kothandaraman and Tianyi Zhou and Ming C. Lin and Dinesh Manocha},
  title = {{AerialBooth: Mutual Information Guidance for Text Controlled Aerial View Synthesis from a Single Image}},
  year = {2023},
  eprint = {2311.15478},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.15478},
}
Figure