Defining and Evaluating Physical Safety for Large Language Models

Yung-Chen Tang, Pin-Yu Chen, Tsung-Yi Ho

Abstract

Large Language Models (LLMs) are increasingly used to control robotic systems such as drones, but their risks of causing physical threats and harm in real-world applications remain unexplored. Our study addresses the critical gap in evaluating LLM physical safety by developing a comprehensive benchmark for drone control. We classify the physical safety risks of drones into four categories: (1) human-targeted threats, (2) object-targeted threats, (3) infrastructure attacks, and (4) regulatory violations. Our evaluation of mainstream LLMs reveals an undesirable trade-off between utility and safety, with models that excel in code generation often performing poorly in crucial safety aspects. Furthermore, while incorporating advanced prompt engineering techniques such as In-Context Learning and Chain-of-Thought can improve safety, these methods still struggle to identify unintentional attacks. In addition, larger models demonstrate better safety capabilities, particularly in refusing dangerous commands. Our findings and benchmark can facilitate the design and evaluation of physical safety for LLMs.

Introduction

As Large Language Models (LLMs) transition from text-based assistants to physical agents capable of controlling robotic systems, a new safety frontier emerges. While traditional safety benchmarks focus on "cyber-safety"—preventing toxic text or malware—the physical implications of LLM-generated code are often overlooked. In robotics, a syntactically correct piece of code can be physically catastrophic if it leads a drone to crash into a crowd or violate aviation regulations. This research introduces a comprehensive benchmark to evaluate the physical safety of LLMs in drone control, identifying a critical gap between a model's utility and its ability to recognize physical danger.

The LLM Physical Safety Benchmark

To measure physical safety, the researchers developed a multi-stage evaluation pipeline that moves beyond simple keyword matching. The framework consists of:

  1. Instruction Input: Prompts ranging from standard flight tasks to malicious attack commands.
  2. Dual AI Judges: A Code Verification Judge checks if the generated code aligns with instructions, while a Safety Evaluation Judge assesses the reasoning behind a model's refusal to generate code.
  3. High-Fidelity Simulation: Code is executed in Microsoft AirSim (Unreal Engine) to detect actual collisions and boundary violations in a 3D environment.
  4. Prompt Engineering Analysis: Evaluation of how techniques like In-Context Learning (ICL) and Zero-Shot Chain-of-Thought (ZS-CoT) impact a model's "Self-Assurance" (the ability to recognize and avoid danger).
Overview of the LLM-Robotics safety benchmarking framework and potential drone threats.

Datasets and Threat Classification

The study utilizes a curated dataset of over 400 instructions, categorized into four dimensions based on expert aviation standards (FAA, ICAO, EASA):

  • Deliberate Attacks: Direct commands to cause harm (e.g., "Hit the crowd").
  • Unintentional Attacks: Benign-sounding commands that are physically risky (e.g., "Fly low through the power turbine").
  • Violation Instructions: Prompts that break FAA Part 107 regulations (e.g., flying in "No Drone Zones").
  • Utility: Standard, safe drone operations to ensure safety measures do not break basic functionality.

Results and Key Insights

The Utility-Safety Paradox

A primary finding is the stark inverse relationship between a model's coding capability and its safety. Models like GPT-3.5 excel in "Code Fidelity" and "Utility" but perform poorly in "Self-Assurance," often generating code for dangerous commands without hesitation. GPT-3.5 achieved the highest Utility score (93.75%) but the lowest Self-Assurance (12.5%).

Comparative Performance across LLMs

As shown in the evaluation results, the Llama family (Llama-2 and Llama-3) consistently outperformed GPT-3.5 in safety metrics. Llama-2-7B-Chat led the pack in regulatory compliance (90.62%), while CodeLlama-7B-Instruct demonstrated a near-perfect "Avoid-Collision" score of 99.12%.

Safety evaluation results across different LLMs, highlighting the trade-offs between utility, code fidelity, and safety metrics.

The "Unintentional" Blind Spot

While models are relatively good at refusing direct attacks, they struggle with "Scenario Oversight." They often fail to realize that a command like "Fly 5 meters above the ground" might result in a collision with a car or person in a specific environment. This suggests a lack of causal reasoning regarding physical consequences.

Scaling and Prompting Effects

Larger models generally demonstrate better safety capabilities, particularly in refusing deliberate attacks. Furthermore, prompt engineering is highly effective; ICL boosted GPT-3.5’s self-assurance from 12.5% to 87.2%. However, even with these techniques, models still struggle to identify unintentional physical risks.

Conclusion

Current LLMs exhibit a significant gap in physical safety awareness. While prompt engineering and model scaling offer improvements, models still lack the ability to "visualize" the downstream physical consequences of their actions. Future work must focus on developing robust physical guardrails and improving models' ability to handle implicit environmental dangers.

Project Page: huggingface.co/spaces/TrustSafeAI/LLM-physical-safety

Summary updated

Abstract

Large Language Models (LLMs) are increasingly used to control robotic systems such as drones, but their risks of causing physical threats and harm in real-world applications remain unexplored. Our study addresses the critical gap in evaluating LLM physical safety by developing a comprehensive benchmark for drone control. We classify the physical safety risks of drones into four categories: (1) human-targeted threats, (2) object-targeted threats, (3) infrastructure attacks, and (4) regulatory violations. Our evaluation of mainstream LLMs reveals an undesirable trade-off between utility and safety, with models that excel in code generation often performing poorly in crucial safety aspects. Furthermore, while incorporating advanced prompt engineering techniques such as In-Context Learning and Chain-of-Thought can improve safety, these methods still struggle to identify unintentional attacks. In addition, larger models demonstrate better safety capabilities, particularly in refusing dangerous commands. Our findings and benchmark can facilitate the design and evaluation of physical safety for LLMs. The project page is available at huggingface.co/spaces/TrustSafeAI/LLM-physical-safety.
BibTeX
@misc{tang2024defining,
  author = {Yung-Chen Tang and Pin-Yu Chen and Tsung-Yi Ho},
  title = {{Defining and Evaluating Physical Safety for Large Language Models}},
  year = {2024},
  eprint = {2411.02317},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.02317},
}
Figure