Signs of Wear: Evaluating Intelligent Speed Assistance Under Synthetic Adverse Conditions

Johannes Fabian Hahn, Mao Shan, Julie Stephany Berrio Pere

TL;DR

Researchers have developed ISA-Wear, a novel dataset that uses generative AI to simulate realistic physical wear—like rust, mud, and overgrown vegetation—on speed-limit signs. Their evaluation reveals a critical vulnerability in Intelligent Speed Assistance (ISA) systems: visual degradation breaks the detection stage long before it prevents a model from actually reading the numbers.

What Problem Does This Paper Address?

Intelligent Speed Assistance (ISA) systems are now mandatory for new vehicles in the EU, relying heavily on cameras to "read" the road. While standard benchmarks suggest traffic sign recognition is a "solved" problem (with models often exceeding human accuracy), these datasets primarily feature well-maintained signs in clear conditions.

In the real world—especially on rural roads—signs are rarely pristine. They fade in the sun, get covered in road spray, or disappear behind growing branches. Collecting a large-scale, controlled dataset of these "worn" signs is incredibly difficult because these conditions are rare and often entangled with complex lighting or viewpoints. This paper addresses the lack of realistic "wear and tear" benchmarks by using generative image editing to create a controlled environment for testing robustness.

Methods

The authors employed a sophisticated "synthetic-to-real" approach:

  1. Generative Editing: They used Gemini 3 Pro Image, an instruction-based multimodal model, to modify real street-level frames. By providing text prompts, they added five types of wear (snow, rust, vegetation, faded paint, and mud) at three severity levels.
  2. Scene Adaptation: Unlike simple filters, the generative model adapted the surrounding scene (e.g., adding snow to the ground when adding it to the sign) while maintaining the original camera viewpoint.
  3. Rigorous Quality Control: Generative models can be "creative" in ways that ruin an experiment (e.g., moving the sign or changing the number). The authors implemented a two-stage filter: a geometric consistency check (comparing the red rim of the sign using an ellipse-fitting algorithm) and a manual review to ensure the sign's identity remained intact.
  4. Perceptual Scoring: Instead of relying on simple pixel-wise metrics like PSNR, they used LPIPS (Learned Perceptual Image Patch Similarity) to measure the "strength" of the wear on a common perceptual scale.

Datasets

  • Source Data: The study started with GTSIGN-220, a dataset of Mapillary street-level imagery.
  • ISA-Wear (New): The resulting dataset contains 883 weathered scenes paired with their original "clean" counterparts. It focuses specifically on circular German speed limit signs.
  • Evaluation Set: To ensure failures were caused by wear and not poor base performance, they only evaluated the 716 frames where the original, clean sign was correctly recognized by the baseline pipeline.

Key Insights and Ideas

  • Detection is the Bottleneck: The most significant "aha moment" is that wear hits the detector first. In a standard "detect-then-read" pipeline, the system often fails to even register that a sign exists, even if the numbers on that sign are still technically legible to a human (or a classifier).
  • Generative AI as a Robustness Tool: The paper demonstrates that modern generative models are now high-fidelity enough to serve as "digital twins" for safety-critical testing, allowing researchers to isolate specific variables (like rust) while keeping the rest of the scene constant.
  • The "Silent Failure" Problem: Missed detections are "silent failures"—the system simply shows no limit, and the driver may not notice. This is arguably more dangerous than an "invalid" read which a system might be able to flag as uncertain.

Key Results

  • Recall Collapse: The YOLOv8 detector, which found 98.3% of signs when clean, only found 62% of the worn signs.
  • Failure Attribution: Missed detections accounted for 78–87% of all failed speed-limit reads.
  • Classification Resilience: If the detector did manage to find the sign, the readers were surprisingly robust. A ViT-based classifier could still read 91% of the detected worn signs.
  • Severity Thresholds: Using LPIPS, the authors found that readability typically drops to 50% when the perceptual distance (severity $d$) reaches approximately 0.3.

Limitations and Future Work

The authors acknowledge several areas for further exploration:

  • Synthetic Gap: While the edits look realistic, they are still synthetic. Future work needs to validate if these results perfectly mirror performance on naturally worn signs.
  • Size Bias: The generative editor struggled with very small signs (often redrawing them larger), meaning the dataset is slightly biased toward larger, easier-to-see signs.
  • Temporal Logic: This study looked at static frames. Real ISA systems use video tracking across multiple frames, which might provide more opportunities to "catch" a worn sign as the vehicle approaches.
  • Scope: The study is limited to daytime imagery and German speed signs; expanding to different regions and lighting conditions (like night or glare) is a logical next step.

Summary updated

Abstract

Intelligent Speed Assistance (ISA) systems rely on cameras to recognize speed-limit signs, yet existing benchmarks underrepresent the weathered, dirty, and overgrown signs commonly encountered on rural roads. We use generative image editing to introduce different forms of wear into real street-level images of speed-limit signs. The resulting ISA-Wear dataset contains paired original and edited images. Guided by text instructions, a multimodal image-editing model adds snow, rust, vegetation, faded paint, or mud to the speed-limit sign in a street-level frame at three severity levels. Where appropriate, the surrounding scene is adapted as well, and the viewpoint is kept. Every edited frame has an unedited counterpart, and we only evaluate frames whose original is recognized correctly, so any failure can be attributed to the wear. Since the model alters the sign's geometry in 21\% of the edits, all edits are filtered by a geometric check and a manual review. Severity is measured as the LPIPS distance between the original and the edited sign plate. On 716 worn frames, we find that in detect-then-read pipelines, wear breaks detection before it breaks reading. The detector finds only 62\% of the worn signs, even though it finds every one of them in the original frames. Missed detections account for 78--87\% of failed reads, a pattern that a zero-shot detector shares. Given the annotated sign box, as in the crop-based benchmarks, a classifier still reads 78\% of the worn signs, but only 56\% when it has to rely on the detector
BibTeX
@misc{hahn2026signs,
  author = {Johannes Fabian Hahn and Mao Shan and Julie Stephany Berrio Pere},
  title = {{Signs of Wear: Evaluating Intelligent Speed Assistance Under Synthetic Adverse Conditions}},
  year = {2026},
  eprint = {2611.20382},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20382},
}
Figure