Introduction
Monitoring energy facilities for liquefied natural gas (LNG) emissions is a high-stakes task for both safety and economic reasons. Traditionally, facilities rely on infrared (IR) cameras because they are highly sensitive to the thermal signatures of gas. However, IR images often lack "context"—the structural details of the surrounding environment—making it difficult to pinpoint exactly where a leak is occurring. While visible-light (VI) cameras provide rich context, they struggle to "see" the gas itself. Furthermore, many existing detection systems rely on "background subtraction," a technique that fails if the camera shakes or if the system is mounted on a mobile platform. The authors propose the MFLNGD (Multimodal Fusion-based LNG Detection) framework to create a stable, portable, and precise detection system that leverages the strengths of both sensor types.
Methodology
The MFLNGD framework follows a multi-stage pipeline designed to integrate disparate sensor data efficiently.
- Image Registration: Since VI and IR cameras have different perspectives, the first step is alignment. The system uses Normalized Total Gradient (NTG) as a target function to match images pixel-by-pixel in an unsupervised manner, ensuring the "warped" VI image overlaps perfectly with the IR image.
- Feature Extraction: Standard deep learning backbones, such as ResNet-18 or VovNet-19, extract features from both the IR and VI streams.
- Fourier Transformer Fusion: This is the core innovation. Standard Vision Transformers (ViTs) are computationally expensive. The authors implemented a Fourier Transformer (FT), which replaces the standard multi-head attention mechanism with a fast Fourier transform. This allows the model to fuse global information from both sensors efficiently without massive memory overhead.
- Detection: Fused features are passed through a Feature Pyramid Network (FPN) and a Faster R-CNN detector to localize LNG plumes.
Experimental Setup
The study utilized a custom dataset from an industrial natural gas facility, consisting of 5,352 video frames (512 × 640 resolution) in both IR and VI formats. Evaluation was conducted using two splits:
- Scene Dependence: A random 70/30 split to test general performance.
- Scene Independence: A rigorous test where the dataset is split by recording scenarios, evaluating how well the model generalizes to entirely new environments.
Results and Discussion
The results confirm that while IR is the primary sensor for gas detection, the structural context from VI images significantly reduces false positives and improves localization.
Key findings include:
- Multimodal Superiority: Using both VI and IR with the Fourier Transformer (R18 backbone) achieved an overall mAP of 27.16, compared to just 11.21 for VI-only and 10.55 for IR-only.
- Fusion Operator Impact: The Fourier Transformer (FT) significantly outperformed simple embedding-based fusion, improving the overall AP50 from 55.02 to 65.20 for the R18 backbone.
- Backbone Scaling: Upgrading to VovNet-19 (V19) yielded the best results in "Scene Dependence," reaching 51.94 mAP.
- Generalization Challenges: A significant performance drop was observed in "Scene Independence" tests (e.g., mAP dropping from ~51 to ~10), highlighting the difficulty of recognizing gas plumes in unfamiliar environments.
Conclusion
The MFLNGD framework demonstrates that global context, captured efficiently via a Fourier Transformer, is essential for detecting non-rigid, amorphous objects like gas plumes. While the system shows promise for real-world surveillance, future work will focus on integrating motion analysis and temporal consistency to improve localization and generalization in diverse industrial settings.
Summary updated