Physically Guided TeX Generation for Low-Light Visual Tasks

Daoheng Li, Jianjun Qian, Mengkai Yan, Jian YangarXiv 2026 · arXiv

TL;DR

TeX-Gen is a new framework that disentangles visible and infrared images into physically meaningful Temperature ($T$), Emissivity ($e$), and Texture ($X$) maps to improve low-light vision. By leveraging a novel, physics-based evaluation metric (TeXMetric) and a task-specific tuning module (TeX Tune), it achieves state-of-the-art performance in object detection and segmentation without requiring expensive hyperspectral hardware or pixel-level physical labels.

What Problem Does This Paper Address?

In low-light environments, visible-spectrum cameras struggle with low contrast, while infrared (IR) sensors provide heat signatures but lack fine texture and color. Traditional "image fusion" methods try to combine these modalities, but they often treat the process as a black box, optimizing for human-pleasing appearance or specific task features.

The authors identify a major hurdle: thermal degeneracy. In standard IR images, different combinations of temperature and emissivity can produce identical pixel intensities, making it hard for a model to truly "understand" the scene physics. While hyperspectral thermal imaging can disentangle these factors, the hardware is prohibitively expensive. Furthermore, there is a lack of pixel-level physical annotations (like ground-truth temperature maps) to train supervised models. TeX-Gen aims to bridge this gap by generating physically consistent representations from standard, affordable visible-IR pairs.

Fig. 1. Motivation for TeX-Gen. (a) Appearance-based fusion versus physically guided TeX generation with task-specific tuning. (b) Image-oriented criteria, TeXMetric constraints, and task-guided channel tuning.

Methods

The TeX-Gen framework consists of two primary stages: TeX Generation and TeX Tune.

  1. TeX Generation: This module takes paired visible and IR images and constructs four "multi-view" features online: cross-modal discrepancy, co-activation, spatial attention, and channel attention. These features are routed through HyperConnection gates into three parallel branches that generate the $T, e,$ and $X$ maps.
  2. TeXMetric: To train this generator without physical labels, the authors introduced TeXMetric. It acts as a physics-guided loss function that enforces:
  3. Radiometric Reconstruction: Using the Stefan-Boltzmann law to ensure the generated maps can mathematically reconstruct the original IR observation.
  4. Channel Structure: Using Sobel gradients to ensure structural consistency across channels.
  5. Material Consistency: Enforcing local smoothness and ordering based on visible cues (e.g., similar materials should have similar emissivity).
  6. TeX Tune: Recognizing that "physically perfect" maps might not always be optimal for every downstream task, TeX Tune adds a lightweight adjustment layer. It uses multi-scale guides and local activity gates to refine the maps for specific tasks (like detection) while keeping the main generator frozen. A unique "Task Penalty" ensures these adjustments only happen if they actually improve the task loss relative to the baseline.
Fig. 2. Overview of TeX-Gen. TeX Generation constructs multi-view features online and selects complementary evidence through channel-specific gates to form the baseline TeX Zb under TeXMetric constraints. With the generator frozen, TeX Tune applies task-specific corrections to Zb through Multi-scale Guide, Local Activity Gate, and Region-Aware processing before passing the tuned TeX Za to the complete downstream task network. Task supervision jointly optimizes the channel-tuning modules and task network, whereas the task penalty updates only the channel-tuning modules.

Datasets

The framework was evaluated across several diverse benchmarks:

  • Object Detection: FLIR (driving scenes), LLVIP (low-light pedestrians), and M3FD (traffic and surveillance).
  • Semantic Segmentation: MSRS (nine-class urban scenes).
  • Physical Validation: The HADAR dataset was used specifically to validate the physical accuracy of the generated maps, as it provides reference channels for temperature and emissivity.

Key Insights and Ideas

  • Physics as a Self-Supervision Signal: The core "aha moment" is that the laws of thermodynamics (radiometric transfer) provide enough constraint to disentangle $T, e,$ and $X$ without needing manual pixel-level labels.
  • Disentanglement over Fusion: Instead of just mixing pixels, the model separates the factors of image formation. This leads to more robust representations that are less susceptible to the "thermal degeneracy" that plagues standard fusion.
  • The Task-Quality Gap: The authors confirm that higher "visual quality" scores in fused images don't always correlate with better task performance. TeX Tune explicitly addresses this by allowing task-specific "nudges" to the physical maps.
  • Metric Correlation: Experiments on the HADAR dataset showed a clear correlation: as the TeXMetric score improved (lower error), the downstream detection performance (mAP) increased, validating the metric's utility.

Key Results

TeX-Gen consistently outperformed existing state-of-the-art (SOTA) fusion and task-specific methods:

  • FLIR Detection: Achieved 87.5% mAP50, surpassing the previous best (MVR-MoE) by nearly 2 percentage points.
  • LLVIP Detection: Reached 72.7% mAP50:95, showing superior localization at strict thresholds.
  • MSRS Segmentation: Achieved a mean IoU of 77.33%, a significant improvement over traditional fusion-based segmentation models.
  • Ablation Studies: Removing any component of TeXMetric (radiometric, structural, or material constraints) led to a measurable drop in performance, proving the necessity of the multi-faceted physical guidance.

Limitations and Future Work

  • Image Alignment: The method assumes visible and infrared images are well-registered. While it handles some discrepancy, significant registration errors remain a challenge for all fusion-based methods.
  • Simplified Physics: The radiometric model used is a "normalized bolometric formulation." While effective, it simplifies complex real-world factors like atmospheric interference and non-opaque surfaces.
  • Future Directions: The authors suggest exploring how this physical disentanglement could be applied to dynamic video sequences or integrated into large-scale foundation models for autonomous systems operating in extreme environments.

Summary updated

Abstract

Low-light conditions reduce visible contrast and obscure details, while infrared images lack color and texture, motivating the fusion of infrared and visible information for visual tasks. Existing fusion methods primarily optimize image appearance or task-specific features without explicitly disentangling the coupled factors underlying thermal observations, limiting their physical interpretation under thermal degeneracy. Hyperspectral thermal measurements enable physical decomposition, but they require specialized imaging hardware, while pixel-level physical annotations for supervision or evaluation incur additional costs. We propose TeX-Gen, a physically guided framework for TeX generation from paired visible and infrared images in low-light visual tasks. TeX Generation constructs complementary multi-view features and employs independent branch-specific gates to route cues into the Temperature, Emissivity, and Texture maps. TeXMetric integrates radiometric reconstruction, channel structure, and material consistency to assess the generated maps without requiring pixel-level physical labels. TeX Tune improves performance through task-guided adjustments, jointly optimizing the adjustment modules and task network while keeping the generator frozen. Experiments show that TeX-Gen achieves competitive task performance in low-light environments on the FLIR, LLVIP, M3FD, and MSRS datasets. Experiments on HADAR also examine the association between TeXMetric scores and downstream detection performance.
BibTeX
@misc{li2026physically,
  author = {Daoheng Li and Jianjun Qian and Mengkai Yan and Jian Yang},
  title = {{Physically Guided TeX Generation for Low-Light Visual Tasks}},
  year = {2026},
  eprint = {2611.10665},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10665},
}
Figure