TL;DR
Autonomous vehicles need to reliably detect human traffic controllers (HTCs) in complex construction zones, but existing datasets are scarce and lack diversity. This paper introduces a novel method using diffusion models to generate realistic synthetic HTC images, significantly improving detection accuracy and releasing a new dataset (HTC600) to advance research in this critical area.
What Problem Does This Paper Address?
As autonomous vehicles (AVs) advance towards higher levels of autonomy (Level 3+), they face increasingly challenging real-world scenarios, particularly construction zones where traditional traffic signals are absent. In these situations, AVs must rely on human traffic controllers (HTCs) to navigate safely. However, detecting HTCs is difficult due to a lack of dedicated, diverse datasets. Existing datasets often suffer from limited variations in clothing, poses, and environmental conditions, making it hard to train robust detection models. This paper aims to bridge this gap by creating a comprehensive synthetic dataset to improve HTC detection performance.
Methods
The proposed method involves a two-stage process:
- Synthetic Data Generation:
- Data Collection & Annotation: They start with 600 real images of HTCs, annotating them with keypoints and masks.
- Color Boosting: A novel color boosting technique is applied to diversify the appearance of HTCs. This involves using models like Grounding DINO and SAM2 to identify specific body parts (helmet, vest, pants) and then applying random adjustments to brightness, contrast, saturation, and hue to these regions. This creates variations without altering the structural integrity of the original images.
- Diffusion Model Training (PIDM): They leverage a pre-trained diffusion model called PIDM (Person Image Diffusion Model) and fine-tune it using their color-boosted HTC images and the DeepFashion dataset. This allows the model to generate new HTC images with diverse poses and styles.
- Cut-and-Paste Integration: The generated HTC images are then seamlessly integrated into realistic road background images. This involves:
- Pasteable Region Identification: Identifying drivable areas in background images.
- Vanishing Point-Based Size Adjustment: A key innovation is using the vanishing point of the road to realistically scale the size of the HTC based on its vertical position in the image, mimicking perspective.
- Multi-Instance Integration & Color Harmonization: Multiple HTCs can be placed in a single scene, and their colors are adjusted to match the background for better visual coherence.
- Model Training & Evaluation:
- A large synthetic dataset (19,840 images) is created by combining the generated HTCs with 992 real road background images.
- This synthetic dataset, along with 600 real HTC images, is used to train a YOLOv7 object detection model.
- The trained model is evaluated on a separate test set of 135 real-world images.
Datasets
- HTC600 Dataset: A newly introduced dataset comprising 600 real human traffic controller images. This dataset is publicly available.
- DeepFashion Dataset: Used in conjunction with the HTC600 dataset for training the PIDM diffusion model.
- Synthesized Training Dataset: A large dataset of 19,840 images generated by combining synthetic HTCs with 992 real road background images.
- HTC600 Test Dataset: 135 real-world images used for evaluating the trained model.
Key Insights and Ideas
- Synthetic Data for Domain-Specific Challenges: The paper demonstrates that synthetic data generation, particularly with advanced diffusion models, is a powerful tool for tackling data scarcity in niche domains like HTC detection, which are crucial for AV safety.
- Color Boosting for Appearance Diversity: The color boosting technique is a simple yet effective way to significantly increase the visual diversity of the training data by varying key clothing items without needing entirely new source images.
- Vanishing Point for Realistic Perspective: Using the vanishing point for instance size adjustment is a clever way to introduce realistic perspective scaling, making synthetic scenes more believable and improving model generalization.
- Incremental Improvement is Key: The ablation study shows that each component of their proposed method (pasteable regions, size adjustment, color boosting, generated images, color adjustment) contributes meaningfully to the overall performance gain.
Key Results
- The proposed method, trained on the synthesized HTC600 dataset, achieved an AP50 score of 73.9% on the HTC600 test dataset.
- This represents a significant improvement of 32.9% over the baseline method (41.0% AP50).
- Ablation studies showed consistent performance gains with each added component.
- The model demonstrated robustness across different daylight conditions (outdoor vs. tunnel), with only a marginal performance difference (approx. 2%).
- The YOLOv7 model used for detection can operate in real-time, handling up to three HTCs per image with minimal increase in inference time.
Limitations and Future Work
- Limited Scope of HTC Actions: The current method assumes HTCs control traffic using only their bare hands. It does not account for scenarios where HTCs use tools like traffic control batons or reflective paddles, which are common in real-world settings.
- Future Work: The authors suggest expanding the dataset to include tool-assisted HTC scenarios and developing synthesis methods for generating HTCs using such tools. They also propose exploring model compression techniques to reduce computational load for real-time deployment and investigating additional safety measures to prevent misuse in critical traffic conditions.
Summary updated