TigAug: Data Augmentation for Testing Traffic Light Detection in Autonomous Driving Systems

Y. P. Lu, Dong Wang, Kaifeng Huang, Bihuan Chen, Xin PengarXiv 2025 · DOI · arXiv

TL;DR

TigAug is a new automated tool designed to stress-test and improve traffic light detection in autonomous driving systems. By applying 12 diverse transformations—ranging from heavy fog to rotating the lights themselves—it identifies hidden model weaknesses and uses synthetic data to boost detection accuracy by over 67% through retraining.

What Problem Does This Paper Address?

Traffic light detection (TLD) is a safety-critical component of autonomous driving, yet it remains a "weak link" in testing. While we have plenty of data for sunny days, collecting and labeling images for the "long tail" of driving scenarios—like a lens flare hitting the camera at sunset or a traffic light being slightly tilted—is incredibly labor-intensive and often impossible to do at scale.

Current testing often relies on manual labeling, which doesn't scale. Furthermore, existing data augmentation techniques are usually generic (like simple cropping). This paper addresses the lack of a systematic way to generate diverse, realistic, and labeled traffic light data to find where state-of-the-art models (like YOLO or Faster R-CNN) fail.

Methods

The authors introduce TigAug, which utilizes Metamorphic Testing. The core idea is to take a "source" image with a known label and transform it into a "follow-up" image where the expected output is mathematically predictable.

TigAug implements 12 specific transformations across three families:

  1. Weather: Simulating rain, snow, fog, and lens flares.
  2. Camera: Simulating overexposure, underexposure, and motion blur.
  3. Traffic Light Properties: This is the most novel part. TigAug can programmatically change a light's color (e.g., red to green), move its position, add new lights to a scene, rotate them (simulating horizontal vs. vertical mounting), or scale them (simulating distance).

To solve the "test oracle" problem (knowing if the model is right without a human checking), they use Metamorphic Relations:

  • Consistency Relation: Weather and camera tweaks shouldn't change the light's position or state. If the model fails here, it's a bug.
  • Transformation-Specific Relation: If TigAug changes a red light to green, the model’s output must change to green. If it still sees red, the model is hallucinating based on context rather than visual evidence.

Datasets

The researchers evaluated TigAug using two industry-standard benchmarks:

  • LISA: A dataset of video sequences collected in California containing over 113,000 annotated traffic lights.
  • Bosch: A dataset containing roughly 24,000 annotated traffic lights.

They pre-processed these to remove duplicates and monochrome images, splitting them into standard training, validation, and testing sets.

Key Insights and Ideas

  • The "Context" Trap: One of the most interesting findings is that models often rely too much on the position of a light bulb rather than its color. By using the "Color Change" (CC) transformation and flipping the light upside down, TigAug revealed that models often incorrectly predict "Red" just because a bulb is at the top, even if the pixel data says "Green."
  • Robustness Gaps: Models are surprisingly fragile. A simple motion blur or a slight rotation of the traffic light caused massive drops in mean Average Precision (mAP).
  • The Value of "Unnatural" Data: About 28% of TigAug’s generated images were deemed "unnatural" by human reviewers (e.g., fog so thick the light is invisible). However, the authors found that including this data in retraining did not hurt performance on real-world data; instead, it helped the models generalize better.
  • Efficiency: TigAug can synthesize a new labeled image in just 0.88 seconds, making it feasible for massive-scale data generation.

Key Results

  • Bug Discovery: When original models (YOLOv5, YOLOX, Faster R-CNN, SSD) were tested on TigAug’s augmented data, their performance (mAP) plummeted by an average of 39.8%. This proved that current models are not nearly as robust as their "clean" dataset scores suggest.
  • Performance Boost: By retraining the models with just 20% of the augmented data added to the original training set, the models' performance on the augmented test sets improved by a staggering 67.5%.
  • Retraining Stability: Importantly, retraining on augmented data didn't "break" the models' ability to see normal lights; performance on the original, real-world datasets remained stable or slightly improved.

Limitations and Future Work

The authors acknowledge a few hurdles:

  • Naturalness: While "unnatural" images didn't hurt retraining, they can lead to "false alarms" during testing (e.g., a model is flagged for failing to see a light that is physically obscured by synthetic fog).
  • Parameter Tuning: The transformations currently rely on empirical settings (like a specific "severity" of snow). Future work could involve using AI to automatically find the most "challenging" parameters that break a model.
  • Expansion: The authors plan to expand the transformation set to include more complex scenarios, such as occlusions (lights blocked by trees or trucks) and more diverse lighting conditions.

You can find the source code and experimental data at https://tigaug.github.io/.

Summary updated

Abstract

Autonomous vehicle technology has been developed in the last decades with recent advances in sensing and computing technology. There is an urgent need to ensure the reliability and robustness of autonomous driving systems (ADSs). Despite the recent achievements in testing various ADS modules, little attention has been paid on the automated testing of traffic light detection models in ADSs. A common practice is to manually collect and label traffic light data. However, it is labor-intensive, and even impossible to collect diverse data under different driving environments. To address these problems, we propose and implement TigAug to automatically augment labeled traffic light images for testing traffic light detection models in ADSs. We construct two families of metamorphic relations and three families of transformations based on a systematic understanding of weather environments, camera properties, and traffic light properties. We use augmented images to detect erroneous behaviors of traffic light detection models by transformation-specific metamorphic relations, and to improve the performance of traffic light detection models by retraining. Large-scale experiments with four state-of-the-art traffic light detection models and two traffic light datasets have demonstrated that i) TigAug is effective in testing traffic light detection models, ii) TigAug is efficient in synthesizing traffic light images, and iii) TigAug generates traffic light images with acceptable naturalness.
BibTeX
@misc{lu2025tigaug,
  author = {Y. P. Lu and Dong Wang and Kaifeng Huang and Bihuan Chen and Xin Peng},
  title = {{TigAug: Data Augmentation for Testing Traffic Light Detection in Autonomous Driving Systems}},
  year = {2025},
  eprint = {2507.05932},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.05932},
  howpublished = {arXiv},
}
Figure