TL;DR
TriNetra introduces a reliability-aware framework for RGB-Thermal (RGB-T) surveillance that uses an explicit, hand-designed Thermal Reliability Coefficient (TRC) to dynamically gate sensor fusion. By calculating frame-level reliability from raw image statistics, the system prevents corrupted thermal data from degrading object detection performance, achieving a significant 5.1-point mAP gain over fixed-weight fusion under sensor failure conditions.
What Problem Does This Paper Address?
Surveillance systems often combine visible (RGB) and thermal cameras to achieve "all-weather" perception. Visible cameras fail in the dark or fog, while thermal cameras are susceptible to sensor saturation, motion blur, and "thermal crossover" (when objects and backgrounds reach the same temperature).
Most current RGB-T fusion models use fixed weights or implicitly learned gating. This creates two major issues:
- Modality Corruption: If one sensor fails, its "garbage" data is still mixed into the final prediction, often leading to worse results than using a single sensor alone.
- Lack of Interpretability: Implicit reliability is hidden within neural network weights, making it impossible for human operators or downstream tasks (like tracking) to inspect why a system is failing.
TriNetra addresses these by proposing an explicit, label-free reliability score that controls the fusion process in real-time.
Methods
The core of the framework is the Thermal Reliability Coefficient (TRC), a scalar between 0 and 1 calculated for every frame pair.
The TRC Calculation
As shown in Figure 2, the TRC aggregates five distinct image cues:
- Contrast: Shannon entropy of the thermal histogram.
- Gradient (Sharpness): Variance of the Laplacian to detect blur.
- Saturation: The percentage of pixels at extreme intensity levels.
- Mutual Information: Statistical dependence between the RGB and thermal frames.
- Alignment: Reprojection error (if available).
TRC-Gated Architecture
The framework uses a dual-branch YOLOv8m detector. To maintain high performance on visible data, the authors kept the visible branch, neck, and head frozen (pre-trained on COCO). The thermal branch is a "warm-started" copy that is trained alongside new gated fusion blocks.
As illustrated in Figure 3, the TRC ($r_T$) acts as a gate: when $r_T$ is high, thermal features are added to the visible stream; when $r_T$ is low (indicating corruption), the thermal contribution is suppressed, and the model falls back to the reliable visible branch.
The TRC is also passed to a ByteTrack-based tracker. In the second stage of association (matching low-confidence detections), the TRC is used to inflate the association cost, requiring a closer motion match when the sensors are deemed unreliable.
Datasets
- M3FD (Multi-spectral Multi-scale Multi-scene Failure Case Dataset): Used for primary evaluation, containing 4,200 pixel-aligned RGB-T pairs across six classes (person, car, bus, motorcycle, truck, lamp).
- Synthetic Tracking Benchmark: 30 seeded sequences with exact ground truth used to test the tracker under controlled degradation.
- M3FD Clips: 14 assembled video clips (378 frames) used for label-free tracking proxies and stress testing.
Key Insights and Ideas
- Explicit vs. Implicit Reliability: By moving reliability estimation outside the black box of the neural network, the authors created a signal that can be plotted, inspected, and used by non-neural components like rule-based behavior analyzers.
- Frozen Visible Path: Leveraging a frozen, COCO-pretrained visible backbone ensures the model doesn't "forget" how to detect objects in clear visible light while learning to incorporate thermal data.
- Graceful Degradation: The system doesn't just fail when a sensor does; it adaptively shifts its reliance. The "gate ablation" study proved that the adaptive gate significantly outperforms "full thermal admission" when the thermal sensor is saturated.
- Label-Free Cues: The TRC doesn't require labeled "failure" examples for training; it relies on fundamental image processing metrics, making it easier to deploy in new environments.
Key Results
- Detection Performance: On the M3FD test split, TriNetra achieved 70.0% mAP@50 and 47.8% mAP@50:95.
- Robustness to Corruption: When the thermal stream was artificially corrupted (blur/saturation), the adaptive TRC gate maintained 43.6% mAP@50, compared to just 38.4% for a fixed-weight gate and 29.2% for full thermal admission.
- Reliability Accuracy: The TRC score stayed stable at ~0.78 on clean frames and dropped predictably to 0.53 under full thermal saturation, confirming it effectively tracks sensor health.
- Tracking: While the TRC-gated association performed on par with standard ByteTrack, it provided a framework for "Low-TRC" flags in behavior analysis, alerting operators when evidence for a track is weak.
Limitations and Future Work
- Real-time Constraints: The current implementation runs at 7.2 FPS on a laptop GPU, which is below the threshold for real-time surveillance.
- Misalignment Detection: The current TRC cues are relatively insensitive to RGB-Thermal misregistration (shifting). The authors suggest that estimating alignment error explicitly is a necessary next step.
- Hand-set Constants: The weights for the five TRC cues were hand-set. Future work could involve learning these weights or re-weighting them dynamically.
- Tracking Data: The tracking evaluation was limited by the lack of long, annotated RGB-T video sequences; the authors plan to annotate a dedicated tracking set for more robust identity accuracy measurements.
Summary updated