TL;DR
Vision-Language Models (VLMs) frequently trigger false alarms in construction safety tasks because they rely on "co-occurrence priors"—the mere presence of a worker and a machine—rather than actual spatial proximity. This study uses mechanistic interpretability to reveal that while models internally encode the correct spatial relations, their final decisions are hijacked by learned biases, a failure that can be partially mitigated by "steering" the model using its own internal evidence.
What Problem Does This Paper Address?
Struck-by incidents involving heavy machinery like excavators are a leading cause of construction fatalities. While VLMs are being deployed to automate safety monitoring, they suffer from a "cry wolf" problem: they flag almost any scene containing both a worker and an excavator as a hazard, regardless of the actual distance between them.
The researchers sought to diagnose exactly where this failure occurs. Does the model fail to see the objects (perception failure)? Does it see them but fail to understand their spatial relationship (relation failure)? Or does it understand the relationship but ignore it in favor of a "worker + excavator = danger" bias (decision failure)?

Methods
The authors employed mechanistic interpretability—a field of AI research that looks "under the hood" of neural networks to understand how they process information. Their approach included:
- Counterfactual Image Generation: They built a sophisticated pipeline using Grounding DINO and SAM 2.1 to create 355 pairs of images. In each pair, the background and excavator are identical, but a single worker is digitally moved from a "near" (hazardous) position to a "far" (safe) position.
- Linear Probing: They trained small "probes" on the internal activations of the VLMs (Qwen2.5-VL and InternVL3.5) to see if information about object presence and spatial distance was actually being represented in the hidden layers.
- Causal Interventions: They used techniques like activation patching (swapping internal states between "near" and "far" images) and token ablation (deleting specific words from the prompt) to see which parts of the input or internal processing actually drove the final "HAZARD" or "SAFE" answer.
- Relation-Gated Steering: A novel correction method that uses the model's own internal "knowledge" of distance to adjust the final output, effectively forcing the model to "listen" to its visual evidence.
Datasets
- ConstructionSite-10k: A large-scale dataset of real-world construction photos used as the source for the counterfactual pairs.
- Controlled Counterfactual Set: 710 images (355 pairs) created for this study to isolate the effect of spatial proximity.
- Natural Transfer Set: 140 unedited photographs used to verify if the findings on edited images hold true in the real world.
- COCO 2017: Used as a general-purpose dataset for training sparse autoencoders.
Key Insights and Ideas
- The "Aha" Moment: The models aren't blind. Probing revealed that both Qwen and InternVL recognize workers and excavators almost perfectly and encode their spatial relationship internally (AUROC of 0.82 and 0.91). The failure is purely at the decision-making stage.
- The Co-Occurrence Offset: The models have a massive "hazard bias" triggered by the joint presence of the two entities. In Qwen, simply adding a worker to a scene increased the hazard score 2.4 times more than moving that worker from a safe distance into the danger zone.
- Language Overrides Vision: Token ablation showed that the words in the prompt (like "worker" and "excavator") set a high baseline hazard score. When descriptive words were removed, the models defaulted to a generic "HAZARD" answer, suggesting the construction context itself evokes a danger prior that visual evidence struggles to overcome.
- Distributed Bias: The researchers couldn't find a single "bad" neuron or attention head responsible for the false alarms. The bias is a distributed property of how the model maps construction scenes to language.
Key Results
- High False Alarm Rates: In the controlled "far" (safe) scenes, Qwen falsely labeled 74% as hazardous, while InternVL labeled 100% as hazardous.
- Internal Accuracy vs. Output Accuracy: While the models' final answers were often wrong, their internal representations of distance were highly accurate.
- Successful Correction: By using "relation-gated subspace steering"—essentially using the model's internal distance probe to gate a correction signal—the researchers improved Qwen's specificity (the ability to correctly identify safe scenes) from 0.36 to 0.58 without significantly hurting its sensitivity to real hazards.
Limitations and Future Work
- Static Images: The study focused on static photos, but real-world construction safety often depends on motion and temporal context (e.g., is the excavator swinging toward the worker?).
- Single Hazard Type: The research was limited to "struck-by" hazards involving excavators. Future work should explore if these co-occurrence priors exist for other risks, like falls from heights or workers under suspended loads.
- Deployment: The steering mechanism is a research tool; translating this into a "plug-and-play" solution for construction site managers remains a challenge. The authors suggest that future VLM development should prioritize "conditioning" decisions on encoded spatial evidence rather than just improving object recognition.
Summary updated