TL;DR
This paper introduces Fire360, a novel benchmark and dataset of 360° firefighting videos designed to push AI's capabilities in challenging, degraded environments. It highlights significant performance gaps between current AI models and human experts in tasks requiring robust perception, memory, and reasoning under conditions like smoke and low light, emphasizing the need for AI that can truly operate under uncertainty.
What Problem Does This Paper Address?
Modern AI systems, particularly vision-language models (VLMs), excel in controlled environments but falter dramatically when faced with real-world complexities like smoke, poor visibility, and structural damage. These are precisely the conditions firefighters encounter daily, where perception failures can have life-threatening consequences. Existing datasets often use clean imagery or synthetic simulations, failing to capture the nuanced challenges of safety-critical scenarios. This gap means current AI is ill-equipped to assist in high-stakes situations where reliable perception, procedural understanding, and memory are paramount. Fire360 aims to bridge this gap by providing a realistic benchmark to evaluate and improve AI's performance in these demanding conditions.
Methods
The core of this work is the Fire360 dataset, comprising 228 professionally recorded 360° videos of firefighter training drills. These videos capture diverse conditions (day/night, indoor/outdoor) with significant visual degradation (smoke, thermal distortion, low light). The dataset is meticulously annotated with action segments, object locations, and degradation metadata.
To evaluate AI capabilities, Fire360 supports five distinct tasks:
- Visual Question Answering (VQA): Assesses spatial reasoning and understanding of object presence and behavior in degraded 360° scenes.
- Temporal Action Captioning: Tests the ability to generate descriptive captions for firefighter actions over time, even under poor visibility.
- Object Localization under Distortion: Evaluates how well models can detect and locate objects (like firefighter gear) when they are occluded or distorted.
- Safety-Critical Reasoning: Probes the AI's understanding of standard safety procedures and its ability to identify violations.
- Transformed Object Retrieval (TOR): A novel task that challenges models to match a pristine object exemplar to its fire-damaged counterpart in an unpaired scene, testing transformation-invariant recognition and episodic memory.
The evaluation is conducted in a zero-shot or prompted setting, reflecting real-world deployment scenarios where models must generalize to unseen, degraded inputs.
Datasets
The primary dataset introduced is Fire360, containing 228 professionally recorded 360° firefighter training videos. This dataset is unique in its focus on safety-critical, degraded environments and its comprehensive annotations.
Key Insights and Ideas
- The "Degradation Gap" is Huge: Current state-of-the-art AI models, even powerful ones like GPT-4o, exhibit a dramatic drop in performance when faced with realistic firefighting conditions compared to clean environments. This gap is significantly larger than human performance degradation.
- Memory and Transformation are Key Challenges: The novel Transformed Object Retrieval (TOR) task reveals a critical weakness in AI: the inability to reliably recognize objects that have undergone significant physical deformation (e.g., melted helmets). This highlights a deficit in episodic memory and transformation-invariant recognition.
- 360° Video Adds Complexity: The panoramic nature of the data introduces unique challenges, particularly spatial distortion near the poles, which further degrades performance on tasks like object localization and retrieval.
- Real-World Data is Crucial: Synthetic data and clean datasets are insufficient for building AI systems that can operate reliably in safety-critical domains. Fire360 provides a much-needed bridge to real-world complexity.
Key Results
- TOR Performance Gap: Human experts achieve 83.5% accuracy on the TOR task, while GPT-4o scores only 39.8%. This stark difference underscores the difficulty of recognizing transformed objects.
- VQA Performance: Even for VQA, human experts maintain over 80% accuracy across conditions, reaching 91.4% overall. GPT-4o, while performing best among tested models, struggles significantly, with accuracy dropping below 10% in heavily degraded scenes.
- Object Localization: Grounding DINO, a strong object detector, performs well in clean conditions (68.2% IoU) but drops to 22.9% in low-visibility scenes, averaging 38.4% overall. Rectilinear projections offer a slight improvement.
- Safety Reasoning: GPT-4o achieves only 28.9% checklist accuracy on safety reasoning, compared to 94.6% for human experts.
Limitations and Future Work
- Generalization: While Fire360 aims for broad applicability, it was collected at a single training institute. Future work could include diverse institutional and international scenarios to improve generalization.
- Temporal Reasoning in TOR: The current TOR task focuses on single frames. Extending it to multi-frame tracking could enhance robustness to severe distortion and occlusion.
- Model Limitations: The paper identifies several failure modes for current models, including overreliance on surface features, lack of priors for material change, and poor spatial grounding in panoramic space. Future research should focus on addressing these.
- Broader Impact: The authors envision Fire360 supporting applications beyond firefighting, such as medical imaging, disaster response, and manufacturing, by advancing AI's ability to reason under uncertainty and recognize transformed objects.
Summary updated