Abstract
Automated security systems often struggle to distinguish between a person merely near a weapon and the actual carrier in crowded environments. This paper introduces a specialized approach for firearm carrier localization, moving beyond simple object detection to identify human-firearm interactions. By employing a novel deep learning architecture featuring attention mechanisms and saliency-driven reconstruction, the proposed E-HFPL model achieves a 77.8% Average Precision (AP), significantly outperforming general human-object interaction models in complex surveillance scenarios.
The Challenge of Complex Scene Localization
While modern AI can reliably detect a "gun" in isolation, real-world surveillance presents significant hurdles. In crowded public spaces like airports or malls, a person might be standing adjacent to a firearm they aren't holding, or a long rifle might visually overlap with multiple individuals in a 2D camera feed.
Existing Human-Object Interaction (HOI) frameworks frequently fail in these "complex scenes" because they either over-prioritize human pose or lose critical spatial details during data processing. This research addresses the gap by creating a system that explicitly associates a firearm with its specific carrier, regardless of clutter or weapon type.
Methodology: The E-HFPL Architecture
The authors propose the Enhanced Human Firearm Pair Localization (E-HFPL) architecture, which treats the human and the firearm as a single interactive unit.
Paired Bounding Boxes (PBB)
Instead of global image analysis, the model generates pairs of detected humans and firearms. It creates a "union" box encompassing both entities to analyze their spatial relationship.
Attention and Saliency
- Attention-Based Masking: The model utilizes binary "attention channels" (masks) for the human and the weapon. These masks guide the neural network to ignore background noise and focus on the most relevant pixels.
- Saliency-Driven Locality Preservation: To prevent the loss of fine-grained details during standard "pooling" operations, a reconstruction branch is added. This forces the feature extractor to retain enough information to rebuild a saliency map, ensuring spatial integrity.
- Adaptive Average Pooling: This component allows the network to process input pairs of any size or aspect ratio, accommodating everything from compact handguns to long-range rifles.
Datasets and Benchmarking
The study utilizes two primary datasets to validate the model:
- Localization of Firearm Carriers (LFC) Dataset: A new benchmark introduced by the authors containing 3,128 images with 6.5k total instances. Every image features at least one human-firearm interaction, meticulously annotated.
- HICO-DET (Subset): A subset of this well-known HOI dataset was used to demonstrate the model's generalizability across 60 object categories involving humans holding objects.
Key Insights
- Contextual Interaction: The model learns the visual "language" of how people carry weapons, recognizing that the hold itself is the primary clue for safety assessment.
- YCbCr Color Space: The authors discovered that the YCbCr color space outperformed standard RGB. By separating luminance from chrominance, the model focuses on structural details and metallic intensity patterns that are more consistent than raw color.
- "Max-Out" Inference: To prevent "double-counting" threats in overlapping scenes, the system assigns a weapon to the person with the highest probability score, marking other overlapping pairs as non-interacting.
Results and Performance
The E-HFPL model demonstrated a marked improvement over existing baselines. On the LFC dataset, it achieved 77.8% AP, a substantial increase from the 63.1% achieved by standard approaches. It also outperformed established HOI models like iCAN (68.1% AP) and TIN (76.1% AP).
In terms of pure classification—determining if a firearm was "carried" or "not carried"—the model reached an overall accuracy of 93.25%.
Limitations and Future Directions
Despite its success, the model faces challenges in specific scenarios:
- Extreme Distance: Small or distant objects can cause association logic to fail.
- Heavy Occlusion: Carriers partially hidden by obstacles or other people may not be correctly linked to their weapons.
- Dense Crowds: Significant overlap in very dense crowds can occasionally lead to misattribution.
Future research will explore the integration of temporal data from video feeds to resolve these ambiguities by tracking how carriers and weapons move in tandem over time.
Summary updated