TL;DR
TeX-Gen is a new framework that disentangles visible and infrared images into physically meaningful Temperature ($T$), Emissivity ($e$), and Texture ($X$) maps to improve low-light vision. By leveraging a novel, physics-based evaluation metric (TeXMetric) and a task-specific tuning module (TeX Tune), it achieves state-of-the-art performance in object detection and segmentation without requiring expensive hyperspectral hardware or pixel-level physical labels.
What Problem Does This Paper Address?
In low-light environments, visible-spectrum cameras struggle with low contrast, while infrared (IR) sensors provide heat signatures but lack fine texture and color. Traditional "image fusion" methods try to combine these modalities, but they often treat the process as a black box, optimizing for human-pleasing appearance or specific task features.
The authors identify a major hurdle: thermal degeneracy. In standard IR images, different combinations of temperature and emissivity can produce identical pixel intensities, making it hard for a model to truly "understand" the scene physics. While hyperspectral thermal imaging can disentangle these factors, the hardware is prohibitively expensive. Furthermore, there is a lack of pixel-level physical annotations (like ground-truth temperature maps) to train supervised models. TeX-Gen aims to bridge this gap by generating physically consistent representations from standard, affordable visible-IR pairs.
Methods
The TeX-Gen framework consists of two primary stages: TeX Generation and TeX Tune.
- TeX Generation: This module takes paired visible and IR images and constructs four "multi-view" features online: cross-modal discrepancy, co-activation, spatial attention, and channel attention. These features are routed through HyperConnection gates into three parallel branches that generate the $T, e,$ and $X$ maps.
- TeXMetric: To train this generator without physical labels, the authors introduced TeXMetric. It acts as a physics-guided loss function that enforces:
- Radiometric Reconstruction: Using the Stefan-Boltzmann law to ensure the generated maps can mathematically reconstruct the original IR observation.
- Channel Structure: Using Sobel gradients to ensure structural consistency across channels.
- Material Consistency: Enforcing local smoothness and ordering based on visible cues (e.g., similar materials should have similar emissivity).
- TeX Tune: Recognizing that "physically perfect" maps might not always be optimal for every downstream task, TeX Tune adds a lightweight adjustment layer. It uses multi-scale guides and local activity gates to refine the maps for specific tasks (like detection) while keeping the main generator frozen. A unique "Task Penalty" ensures these adjustments only happen if they actually improve the task loss relative to the baseline.
Datasets
The framework was evaluated across several diverse benchmarks:
- Object Detection: FLIR (driving scenes), LLVIP (low-light pedestrians), and M3FD (traffic and surveillance).
- Semantic Segmentation: MSRS (nine-class urban scenes).
- Physical Validation: The HADAR dataset was used specifically to validate the physical accuracy of the generated maps, as it provides reference channels for temperature and emissivity.
Key Insights and Ideas
- Physics as a Self-Supervision Signal: The core "aha moment" is that the laws of thermodynamics (radiometric transfer) provide enough constraint to disentangle $T, e,$ and $X$ without needing manual pixel-level labels.
- Disentanglement over Fusion: Instead of just mixing pixels, the model separates the factors of image formation. This leads to more robust representations that are less susceptible to the "thermal degeneracy" that plagues standard fusion.
- The Task-Quality Gap: The authors confirm that higher "visual quality" scores in fused images don't always correlate with better task performance. TeX Tune explicitly addresses this by allowing task-specific "nudges" to the physical maps.
- Metric Correlation: Experiments on the HADAR dataset showed a clear correlation: as the TeXMetric score improved (lower error), the downstream detection performance (mAP) increased, validating the metric's utility.
Key Results
TeX-Gen consistently outperformed existing state-of-the-art (SOTA) fusion and task-specific methods:
- FLIR Detection: Achieved 87.5% mAP50, surpassing the previous best (MVR-MoE) by nearly 2 percentage points.
- LLVIP Detection: Reached 72.7% mAP50:95, showing superior localization at strict thresholds.
- MSRS Segmentation: Achieved a mean IoU of 77.33%, a significant improvement over traditional fusion-based segmentation models.
- Ablation Studies: Removing any component of TeXMetric (radiometric, structural, or material constraints) led to a measurable drop in performance, proving the necessity of the multi-faceted physical guidance.
Limitations and Future Work
- Image Alignment: The method assumes visible and infrared images are well-registered. While it handles some discrepancy, significant registration errors remain a challenge for all fusion-based methods.
- Simplified Physics: The radiometric model used is a "normalized bolometric formulation." While effective, it simplifies complex real-world factors like atmospheric interference and non-opaque surfaces.
- Future Directions: The authors suggest exploring how this physical disentanglement could be applied to dynamic video sequences or integrated into large-scale foundation models for autonomous systems operating in extreme environments.
Summary updated