Introduction
Generating high-fidelity aerial views from a single ground-level photograph is a formidable challenge in computer vision. While modern text-to-image models can produce generic aerial scenes, they often struggle to preserve the specific semantic details of a provided subject, such as a particular lighthouse or a specific group of friends. Existing methods typically rely on extensive 3D data or struggle with the drastic perspective shifts required for aerial synthesis. HawkI (AerialBooth) introduces a "3D-free" framework that bridges this gap by combining classical geometric techniques with information theory.
Methodology: Bridging Geometry and Information Theory
HawkI utilizes a two-stage approach integrated with a pretrained Stable Diffusion model to achieve realistic perspective transformations without the need for massive 3D datasets.
1. Test-Time Optimization with Inverse Perspective Mapping (IPM)
The framework begins with a brief finetuning step on the input image. To guide the model toward an aerial perspective, the authors incorporate Inverse Perspective Mapping (IPM). Although IPM often produces distorted "top-down" views, these serve as a crucial geometric hint or "compass," breaking the model's inherent bias toward the original ground-level viewpoint.
2. Mutual Information (MI) Guided Inference
During the denoising process, HawkI employs a novel guidance strategy based on Mutual Information. Unlike standard pixel-wise metrics (like L2 distance), MI measures the statistical dependence between image distributions. This allows the model to enforce semantic consistency—ensuring the generated aerial view contains the same content as the input—without being constrained by the original pixel-level layout.
Key Results and Evaluation
The researchers evaluated the framework using two newly curated datasets: HawkI-Syn (500 synthetic images with LLM-generated prompts) and HawkI-Real (139 real-world photos from Unsplash). These datasets span diverse categories, including architecture, natural landscapes, and human actions.
HawkI demonstrates significant qualitative and quantitative improvements over baseline methods like DreamBooth and Aerial Diffusion. It successfully generates images that are both "more aerial" in perspective and more faithful to the original subject's identity. Quantitatively, the method achieves a superior balance between text alignment (CLIP-Score) and image fidelity (DINO/SSCD). Notably, HawkI performs comparably to state-of-the-art 3D-trained models like Zero123++, despite requiring no 3D training data. Furthermore, using HawkI-generated data improved zero-shot action recognition for drones (UAVs) by 7.14%, proving the synthetic images' realism for machine learning applications.
Conclusion and Future Outlook
HawkI proves that the 3D knowledge embedded within 2D generative models can be effectively unlocked for novel view synthesis through classical computer vision "nudges." While the method currently faces limitations regarding hallucination and the need for more precise quantitative camera controls, its success in generating high-fidelity aerial views from single images marks a significant step forward for data augmentation and creative tools.
Summary updated