TL;DR
This paper introduces RGBT-Textile, a new dataset for segmenting textile damage using RGB and thermal (RGBT) imagery, and a novel normalization technique called ThermoFreq. ThermoFreq effectively reduces thermal noise, significantly improving the performance of state-of-the-art RGBT semantic segmentation models on this challenging task, which is crucial for sustainable textile repair.
What Problem Does This Paper Address?
The textile industry faces pressure to reduce waste and embrace sustainability. A key aspect of this is extending the lifespan of garments through repair. Automating the detection of textile damage is a critical step towards this goal, and RGB-Thermal (RGBT) imaging offers a promising approach. Thermal imaging can reveal subtle material properties and damages that might be missed by RGB cameras alone, especially in varying lighting conditions. However, a significant challenge in RGBT semantic segmentation, particularly for textile damage, is the scarcity of dedicated, publicly available datasets. Furthermore, thermal images are prone to noise and variations due to ambient temperature, which can degrade segmentation performance. Existing normalization techniques often fall short in addressing these specific thermal image challenges.
Methods
The authors introduce two main contributions:
- RGBT-Textile Dataset: This dataset was meticulously collected and annotated for textile damage segmentation. It comprises 1441 RGBT image pairs, each with ground-truth segmentation masks for three classes: background, garment, and damage. The data collection involved a specific protocol to capture variations in garment types and damage, including scenarios with and without induced thermal variations.
- ThermoFreq Normalization: This is a novel frequency-domain thermal normalization technique. It works by transforming the thermal image into the frequency domain using a Fourier transform. High-frequency components, often associated with noise, are then selectively filtered or attenuated. The modified frequency domain representation is then transformed back to the spatial domain. This process aims to enhance salient thermal features while mitigating noise, without significantly altering the overall temperature distribution.
Datasets
- RGBT-Textile Dataset: A newly introduced benchmark dataset specifically for textile damage segmentation. It contains 1441 RGBT image pairs with detailed annotations.
- Existing RGBT Datasets: The paper also evaluates their proposed method on six other publicly available RGBT datasets: KAIST Multispectral Pedestrian, LLVIP, MFNet, OSU Thermal Pedestrian, McubeS, and PST900. This allows for a broader assessment of ThermoFreq's effectiveness.
Key Insights and Ideas
- Thermal Imaging's Potential for Textiles: Thermal imaging is not just for detecting heat signatures; it can reveal material properties and subtle damages that are invisible to RGB cameras, making it valuable for textile inspection.
- The Challenge of Thermal Noise: Ambient temperature fluctuations and sensor noise significantly impact the quality of thermal images, posing a major hurdle for accurate semantic segmentation.
- Frequency Domain for Thermal Normalization: By operating in the frequency domain, ThermoFreq can selectively target and suppress noise components that are often problematic in thermal imagery, offering a more sophisticated approach than traditional spatial-domain methods.
- Dataset Specificity Matters: The optimal parameters for thermal normalization (like the tolerance threshold in ThermoFreq) can vary significantly depending on the dataset's characteristics, particularly its temperature range and the nature of the thermal variations.
Key Results
- ThermoFreq Improves Performance: Experiments show that applying ThermoFreq consistently improves the performance of state-of-the-art RGBT segmentation models (CRM_RGBTSeg, MMSFormer, and UNet variants) across various datasets, including the new RGBT-Textile dataset. Improvements in mean Intersection over Union (mIoU) range from 1% to 2.2% compared to baseline models.
- Transformer Models Excel: Transformer-based models (CRM_RGBTSeg, MMSFormer) generally outperform CNN-based models (UNet) by 1-2% in mIoU, demonstrating their superior ability to capture global context and multimodal dependencies.
- RGBT-Textile is Challenging: The RGBT-Textile dataset is moderately challenging, with the "damaged area" class being the most difficult to segment, achieving mIoU values around 0.645-0.675.
- Robustness to Noise: ThermoFreq-enhanced models show greater robustness to synthetic thermal noise and occlusions, with CNN-based models exhibiting larger performance drops when not normalized.
Limitations and Future Work
- Parameter Sensitivity: The effectiveness of ThermoFreq is sensitive to its tolerance threshold parameter (t), which needs to be tuned for different datasets. Datasets with higher thermal temperature ranges benefit from higher 't' values, while those with lower ranges require lower 't' values to avoid losing important information.
- Damage Area Segmentation: While improved, segmenting damaged areas remains the most challenging aspect. Future work could focus on enhancing the normalization technique to better highlight these features or augmenting the dataset with more diverse damage examples.
- Dataset Expansion: The authors plan to expand the RGBT-Textile dataset with more samples and labels to further improve model training and evaluation.
- Broader Applications: Investigating the application of ThermoFreq to other domains where thermal imaging is used for segmentation could be a valuable future direction.
Summary updated