UTA-Sign: Unsupervised Thermal Video Augmentation via Event-Assisted Traffic Signage Sketching

Yuqi Han, Songqian Zhang, Weijian Su, Ke Li, Jiayu Yang, Jinli Suo, Qiang ZhangarXiv 2025 · DOI · arXiv

Abstract

Thermal cameras are essential for nighttime autonomous driving but struggle to capture traffic signage and license plates, which often appear as featureless gray blobs due to uniform temperatures. We present UTA-Sign, a novel unsupervised framework that "sketches" missing signage details into thermal video by leveraging the high-speed edge detection of event cameras. By fusing these modalities without labeled datasets, UTA-Sign enables clear perception of road signs in total darkness, significantly enhancing downstream object detection accuracy.

Introduction: The Thermal "Blind Spot"

Thermal imaging is a staple for nighttime perception because it detects heat radiation rather than visible light. However, it suffers from a significant limitation: objects like traffic signs, roadblock indicators, and license plates often maintain the same temperature as their surroundings. Consequently, they lack the thermal contrast necessary for identification, appearing as indistinct shapes in the imagery.

Neuromorphic event cameras offer a potential solution by capturing high-frequency edge details (changes in light intensity) even in low-light conditions. However, event data is asynchronous, sparse, and noisy, making it difficult to integrate with stable thermal frames. UTA-Sign addresses this by fusing the steady geometry of thermal cameras with the high-detail "flicker" of event data in an unsupervised manner.

Methodology

The UTA-Sign framework consists of two primary modules designed to handle the unique characteristics of thermal and event data:

1. Signage Information Sketching (SIS)

The SIS module utilizes a dual-modal encoder-decoder architecture. It employs a "mask-guided" unsupervised strategy to identify semantic regions (such as signs) within the event stream. These details are then "sketched" onto the corresponding thermal frame, ensuring that the original thermal background is preserved while the critical signage information is restored.

2. Temporal Consistency Correction (TCC)

Because event data is non-uniform and depends on motion, stationary periods can lead to data gaps and flickering in the video output. The TCC module mitigates this by using a Deformable CNN and a Swin-Transformer to align features across multiple frames. It leverages the thermal camera’s steady motion as a temporal reference to "fill in the gaps" of the event data, ensuring a smooth and consistent video stream.

Unsupervised Training via Pseudo Ground Truths: To train the model without manual labels, the authors developed a method to generate Pseudo Ground Truths. By using precise motion estimation from the thermal camera, they warped and aligned asynchronous event signals into a unified time frame, creating high-quality targets for the model to emulate.

Key Results

The performance of UTA-Sign was evaluated against six baseline methods (including Text-IF and Retinexformer) using a dataset of 34 distinct real-world and simulated scenes.

Comparison of raw infrared frames, raw event signals, and the final UTA-Sign augmented results.

Perceptual Quality

UTA-Sign achieved superior scores across multiple metrics, notably a Perceptual Index (PI) of 3.287 (where lower is better) and a Standard Deviation (SD) of 66.080. These results indicate significantly better contrast and detail recovery compared to existing fusion methods.

Downstream Detection Performance

The practical utility of the framework was demonstrated by feeding augmented frames into a YOLOv9 detector trained on the TT100K dataset. In scenarios where thermal-only or standard RGB-fusion methods failed to detect signs, UTA-Sign provided "sketches" clear enough for the detector to accurately localize and classify the signage.

Conclusion

UTA-Sign represents a significant step forward in nighttime autonomous perception. By treating the thermal camera as a geometric "anchor" and the event camera as the "ink" for semantic details, the system overcomes the inherent limitations of thermal imaging for traffic signage. Its unsupervised nature allows for scalable deployment across diverse real-world environments without the need for costly manual labeling. Future work will focus on improving calibration sensitivity and robustness in extreme lighting conditions.

Summary updated

BibTeX
@misc{han2025uta,
  author = {Yuqi Han and Songqian Zhang and Weijian Su and Ke Li and Jiayu Yang and Jinli Suo and Qiang Zhang},
  title = {{UTA-Sign: Unsupervised Thermal Video Augmentation via Event-Assisted Traffic Signage Sketching}},
  year = {2025},
  eprint = {2508.20594},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.20594},
}
Figure