Text2Street: Controllable Text-to-image Generation for Street Views

Jinming Su, Songen Gu, Yiting Duan, Xingyue Chen, Junfeng Luo

TL;DR

Text2Street is a novel framework designed to generate highly controllable street-view images from textual descriptions. By decomposing the generation process into intermediate steps—creating a road map, then an object layout, and finally the RGB image—it overcomes the common failures of general diffusion models in handling precise lane counts, complex intersections, and specific traffic densities.

What Problem Does This Paper Address?

While general text-to-image (T2I) models like Stable Diffusion or Midjourney produce stunning visuals, they struggle with the rigid constraints of specialized domains like autonomous driving. If you ask a standard model for a "street view with exactly 3 lanes and 2 cars," you often get a messy intersection, an incorrect number of lanes, or a random swarm of vehicles.

The authors identify three primary challenges in street-view generation:

  1. Complex Road Topology: Generating accurate intersections and specific lane markings that follow traffic rules.
  2. Diverse Traffic Status: Precisely controlling the number and placement of vehicles and pedestrians.
  3. Variable Weather: Ensuring weather effects (like rain) look realistic and consistent with the scene geometry.
Figure 1. Challenges of text-to-image generation for street views. There are three primary challenges: (1) complex road topology, including road structure in the first row and topological marks in the second row, (2) diverse traffic status, e.g., varying traffic objects in the third row, and (3) various weather conditions like the rainy day in the last row. Note that Reference are original images from nuScenes [3], Stable Diffusion [25]/Midjourney [19]/DALLE3 [2] are tested on their official APIs, and Stable Diffusion∗and Ours are finetuned on nuScenes.

Methods

The core innovation of Text2Street is its "divide and conquer" architecture, which moves away from direct text-to-pixel generation. Instead, it uses a three-stage pipeline:

  1. Lane-aware Road Topology Generator (LRTG): This module takes text describing the road (e.g., "crossing, 3 lanes") and generates a top-down local semantic map. It features a unique "counting adapter" that monitors cross-attention layers to ensure the generated map contains the exact number of lane lines requested.
  2. Position-based Object Layout Generator (POLG): Using the generated road map as a constraint, this module uses a bounding box diffusion strategy to place traffic objects (cars, trucks, etc.). By incorporating the road map via a ControlNet, it ensures that cars are actually placed on drivable surfaces rather than sidewalks.
  3. Multiple Control Image Generator (MCIG): Finally, the road map and object layout are projected from a top-down view to a camera perspective. These projections, along with weather-related text prompts, are fed into a final diffusion model equipped with multiple ControlNets to synthesize the final realistic street-view image.
Figure 2. Framework of Text2Street. We begin by introducing the lane-aware road topology generator, which utilizes textual input to create a local semantic map representing the intricate road topology with lane information. Next, we present the position-based object layout generator, which captures the diversity of traffic status and generate the traffic object layout. Subsequently, the road topology and object layout are projected into the camera’s perspective through pose sampling. Finally, the projected road topology, object layout, and textual weather description are integrated through the multiple control image generator to produce the ultimate street-view image.

Datasets

The researchers utilized the nuScenes dataset, a standard benchmark in the autonomous driving community. It contains 1,000 scenes with rich semantic maps and multi-camera RGB images. For this study, the authors focused on images from the FRONT camera to validate their approach.

Key Insights and Ideas

  • Intermediate Representations are Key: Direct text-to-image is too "black box" for high-precision tasks. By generating a semantic map first, the model gains a "ground truth" geometry to follow.
  • Solving the Counting Problem: Diffusion models are notoriously bad at counting. The "counting adapter" in the LRTG is a clever way to force the model to pay attention to numerical tokens in the prompt.
  • Geometric Consistency: Projecting 2D maps and 3D layouts into the camera's perspective before final image generation ensures that the resulting image is physically plausible.
  • Downstream Utility: The paper demonstrates that these generated images aren't just for show—they can be used to augment real datasets to improve the performance of object detectors like YOLOv5.

Key Results

Text2Street significantly outperformed fine-tuned versions of Stable Diffusion 1.5 and 2.1 across both image-quality and attribute-accuracy metrics:

  • Attribute Accuracy: The model showed a 14.91% improvement in object counting accuracy ($S_{obj}$) and a 4.5% improvement in lane counting accuracy ($S_{lane}$) over the next best method.
  • Image Fidelity: It achieved a lower (better) FID score of 53.92, compared to 63.83 for Stable Diffusion 2.1, indicating higher visual realism.
  • Data Augmentation: Adding 30,000 Text2Street-generated images to the nuScenes training set improved a YOLOv5 detector's mAP from 46.30 to 47.83, proving the synthetic data's value for training AI.
Qualitative comparison showing Text2Street's superior adherence to prompts compared to Stable Diffusion.

Limitations and Future Work

While the results are promising, the paper notes some areas for growth:

  • Single View Focus: The current experiments are limited to the front-facing camera. Real-world autonomous driving requires a consistent 360-degree panoramic view.
  • Temporal Consistency: The framework generates static images. Extending this to generate temporally consistent video sequences would be the next major hurdle for simulation.
  • Image Editing: While the authors show the model can edit existing scenes (e.g., changing weather or adding a car), the complexity of these edits is currently limited by the underlying semantic map's flexibility.

Summary updated

Abstract

Text-to-image generation has made remarkable progress with the emergence of diffusion models. However, it is still a difficult task to generate images for street views based on text, mainly because the road topology of street scenes is complex, the traffic status is diverse and the weather condition is various, which makes conventional text-to-image models difficult to deal with. To address these challenges, we propose a novel controllable text-to-image framework, named \textbf{Text2Street}. In the framework, we first introduce the lane-aware road topology generator, which achieves text-to-map generation with the accurate road structure and lane lines armed with the counting adapter, realizing the controllable road topology generation. Then, the position-based object layout generator is proposed to obtain text-to-layout generation through an object-level bounding box diffusion strategy, realizing the controllable traffic object layout generation. Finally, the multiple control image generator is designed to integrate the road topology, object layout and weather description to realize controllable street-view image generation. Extensive experiments show that the proposed approach achieves controllable street-view text-to-image generation and validates the effectiveness of the Text2Street framework for street views.
BibTeX
@inproceedings{su2024text,
  author = {Jinming Su and Songen Gu and Yiting Duan and Xingyue Chen and Junfeng Luo},
  title = {{Text2Street: Controllable Text-to-image Generation for Street Views}},
  booktitle = {ICPR},
  year = {2024},
}
Figure