TL;DR
Text2Street is a novel framework designed to generate highly controllable street-view images from textual descriptions. By decomposing the generation process into intermediate steps—creating a road map, then an object layout, and finally the RGB image—it overcomes the common failures of general diffusion models in handling precise lane counts, complex intersections, and specific traffic densities.
What Problem Does This Paper Address?
While general text-to-image (T2I) models like Stable Diffusion or Midjourney produce stunning visuals, they struggle with the rigid constraints of specialized domains like autonomous driving. If you ask a standard model for a "street view with exactly 3 lanes and 2 cars," you often get a messy intersection, an incorrect number of lanes, or a random swarm of vehicles.
The authors identify three primary challenges in street-view generation:
- Complex Road Topology: Generating accurate intersections and specific lane markings that follow traffic rules.
- Diverse Traffic Status: Precisely controlling the number and placement of vehicles and pedestrians.
- Variable Weather: Ensuring weather effects (like rain) look realistic and consistent with the scene geometry.
Methods
The core innovation of Text2Street is its "divide and conquer" architecture, which moves away from direct text-to-pixel generation. Instead, it uses a three-stage pipeline:
- Lane-aware Road Topology Generator (LRTG): This module takes text describing the road (e.g., "crossing, 3 lanes") and generates a top-down local semantic map. It features a unique "counting adapter" that monitors cross-attention layers to ensure the generated map contains the exact number of lane lines requested.
- Position-based Object Layout Generator (POLG): Using the generated road map as a constraint, this module uses a bounding box diffusion strategy to place traffic objects (cars, trucks, etc.). By incorporating the road map via a ControlNet, it ensures that cars are actually placed on drivable surfaces rather than sidewalks.
- Multiple Control Image Generator (MCIG): Finally, the road map and object layout are projected from a top-down view to a camera perspective. These projections, along with weather-related text prompts, are fed into a final diffusion model equipped with multiple ControlNets to synthesize the final realistic street-view image.
Datasets
The researchers utilized the nuScenes dataset, a standard benchmark in the autonomous driving community. It contains 1,000 scenes with rich semantic maps and multi-camera RGB images. For this study, the authors focused on images from the FRONT camera to validate their approach.
Key Insights and Ideas
- Intermediate Representations are Key: Direct text-to-image is too "black box" for high-precision tasks. By generating a semantic map first, the model gains a "ground truth" geometry to follow.
- Solving the Counting Problem: Diffusion models are notoriously bad at counting. The "counting adapter" in the LRTG is a clever way to force the model to pay attention to numerical tokens in the prompt.
- Geometric Consistency: Projecting 2D maps and 3D layouts into the camera's perspective before final image generation ensures that the resulting image is physically plausible.
- Downstream Utility: The paper demonstrates that these generated images aren't just for show—they can be used to augment real datasets to improve the performance of object detectors like YOLOv5.
Key Results
Text2Street significantly outperformed fine-tuned versions of Stable Diffusion 1.5 and 2.1 across both image-quality and attribute-accuracy metrics:
- Attribute Accuracy: The model showed a 14.91% improvement in object counting accuracy ($S_{obj}$) and a 4.5% improvement in lane counting accuracy ($S_{lane}$) over the next best method.
- Image Fidelity: It achieved a lower (better) FID score of 53.92, compared to 63.83 for Stable Diffusion 2.1, indicating higher visual realism.
- Data Augmentation: Adding 30,000 Text2Street-generated images to the nuScenes training set improved a YOLOv5 detector's mAP from 46.30 to 47.83, proving the synthetic data's value for training AI.
Limitations and Future Work
While the results are promising, the paper notes some areas for growth:
- Single View Focus: The current experiments are limited to the front-facing camera. Real-world autonomous driving requires a consistent 360-degree panoramic view.
- Temporal Consistency: The framework generates static images. Extending this to generate temporally consistent video sequences would be the next major hurdle for simulation.
- Image Editing: While the authors show the model can edit existing scenes (e.g., changing weather or adding a car), the complexity of these edits is currently limited by the underlying semantic map's flexibility.
Summary updated