Intelligent Disobedience in Autonomous Agents

18 concepts10 papers2 proposals

When we train autonomous systems—whether they are self-driving cars, robotic assistants, or digital GUI agents—our default instinct is to optimize for perfect obedience. We want the agent to execute user commands precisely as stated.

However, in real-world deployment, blind obedience is a severe safety hazard. If a navigation system instructs an autonomous vehicle to turn right where no road exists, or if a visually impaired handler mistakenly commands a guide robot to step into oncoming traffic, a perfectly obedient agent will cause a catastrophe.

To solve this, researchers are developing frameworks for intelligent disobedience: the capacity of an autonomous agent to evaluate human instructions against real-world constraints and deliberately refuse to execute commands that would lead to hazardous outcomes.

The Failure of Blind Obedience

Traditional end-to-end models excel at mapping high-level instructions to actions under normal conditions. However, they lack the "common sense" required to handle erroneous, impossible, or dangerous commands.

As Carton et al. et al. demonstrated in Evaluating Robustness over High Level Driving Instruction for Autonomous Driving, state-of-the-art autonomous driving models frequently exhibit blind obedience, prioritizing external navigation commands over immediate physical realities—such as attempting to turn directly into a solid wall because a global planner instructed them to do so.

This vulnerability stems from a fundamental priority conflict: the agent treats the human command as an absolute truth rather than a conditional request. To achieve true safety, the system must treat environmental constraints and safety protocols as unyielding boundaries, while treating human instructions as flexible inputs that can be overridden. As highlighted in the Reports on the 2018 AAAI Spring Symposium Series by Amato et al. et al., high-stakes environments necessitate that autonomous systems possess the authority to proactively intervene and override human control to prevent catastrophic failures.

The Architecture of Principled Refusal

How do we design an agent that knows when and why to say no? Recent frameworks achieve this by decoupling instruction parsing from action execution, introducing an intermediate validation or arbitration layer.

[Human Command] ──> [Safety Arbitration Layer] <── [Environmental Sensors]
                             │
            ┌────────────────┴────────────────┐
            ▼                                 ▼
   [Safe & Feasible?]                [Unsafe / Impossible?]
            │                                 │
            ▼                                 ▼
    Execute Action                    Reasoned Refusal / Immobilization

1. Safety-First Arbitration Logic

Instead of directly translating text or voice commands into control signals, modern multi-modal architectures route instructions through an explicit arbitration gate. In LMDrive: Closed-Loop End-to-End Driving with Large Language Models, Shao et al. et al. implemented a safety-first arbitration logic. The model is trained to evaluate user requests against real-time multi-modal sensor data (like LiDAR and cameras) and traffic laws. If a user asks for an illegal or dangerous maneuver, the arbitration logic overrides the command to maintain operational safety.

2. Feasibility Detection via Trajectory Projection

Another approach relies on predicting the future consequences of an instruction before executing it. In iMotion-LLM: Motion Prediction Instruction Tuning, Felemban et al. et al. integrated Large Language Models with motion prediction. By conditioning trajectory generation on textual instructions, the system can simulate the future path implied by a command. If the feasibility detection mechanism determines that the generated trajectory intersects with an obstacle or violates safety boundaries, the system rejects the instruction entirely.

3. Grounding via Auxiliary Tasks

To prevent agents from ignoring their surroundings in favor of high-level goals, researchers ground the agent's internal representations in physical reality. Carton et al. et al. found that training autonomous agents with an auxiliary semantic segmentation task significantly improves their environmental understanding. By forcing the network to explicitly label obstacles and drivable areas, the agent develops the spatial awareness necessary to recognize when a command contradicts physical reality.

Expanding the Refusal Paradigm

The necessity of intelligent disobedience extends beyond physical vehicles into assistive robotics and digital environments.

Assistive Robotics and Guide Dogs

In automated care and assistive navigation, intelligent disobedience is a core requirement for user autonomy. In The Seeing-Eye Robot Grand Challenge: Rethinking Automated Care, Mirsky et al. et al. proposed using biological seeing-eye dogs as a blueprint for AI. Guide dogs routinely disobey commands that endanger their handler.

Without this capability, assistive machines remain hazardous. DeFazio et al. et al. noted in Seeing-Eye Quadruped Navigation with Force Responsive Locomotion Control that standard robotic guides often lack the high-level reasoning required to refuse directional inputs that lead toward immediate hazards. When designed correctly, however, this feature directly fosters user trust. In a study on user preferences for robotic guide dogs, Cohav et al. et al. in Do Looks Matter? Exploring Functional and Aesthetic Design Preferences for a Robotic Guide Dog discovered that visually impaired users actively expect and desire an autonomous refusal mechanism to ensure their safety in complex environments.

Preventing Over-Execution in Digital Agents

In the digital realm, intelligent disobedience manifests as a defense against "over-execution"—the tendency of LLM-based GUI agents to blindly execute ambiguous or changing instructions until they fail. To counter this, Cheng et al. et al. introduced OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agents. By training agents to evaluate their own confidence levels at each step, the system can dynamically pause and refuse to proceed without human clarification when an instruction becomes ambiguous or the environment changes unexpectedly.

Ethical Boundaries and Immobilization

Intelligent disobedience also serves as a critical line of defense against harmful or biased instructions. In Robots Enact Malignant Stereotypes, Hundt et al. et al. demonstrated that when robotic manipulation systems are given instructions embedded with toxic or stereotypical biases, standard mitigation strategies fail. The authors argue that complete immobilization—a absolute refusal to act on a problematic command—provides a far safer and more robust outcome than attempting to execute an unsafe or unethical instruction.

The Shift Toward Reasoned Interaction

When an agent refuses a command, it should not simply shut down; it must communicate its reasoning to maintain a collaborative loop with the human operator.

Architectures like DriveMLM, developed by Cui et al. et al. in DriveMLM: aligning multi-modal large language models with behavioral planning states for autonomous driving, achieve this by aligning the internal states of Multimodal Large Language Models (MLLMs) with behavioral planning states. When a user command conflicts with safety protocols, the system does not just ignore the input; it provides a reasoned refusal, explaining exactly which environmental constraint or traffic rule prevented the action.

As software engineering roadmaps for responsible AI mature (such as the framework proposed by Lu et al. et al. in Towards a Roadmap on Software Engineering for Responsible AI), the balance between human override capabilities and autonomous safety interventions will become a standardized architectural layer. By treating disobedience not as a system flaw but as a necessary safety mechanism, we can build autonomous agents that are truly robust, responsible, and trustworthy.

Key Foundational Work

Evaluating Robustness over High Level Driving Instruction for Autonomous Driving (2021) — contributed 2 concept elements to this direction.
The Seeing-Eye Robot Grand Challenge: Rethinking Automated Care (2021) — contributed 1 concept elements to this direction.
Reports on the 2018 AAAI Spring Symposium Series (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_intelligent_disobedience_dcb55544,
  title = {intelligent disobedience},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/intelligent-disobedience/rdr_dcb55544d3be4a968165c6637f1a1590},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_intelligent_disobedience_dcb55544_v1,
  title = {intelligent disobedience},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/intelligent-disobedience/rdr_dcb55544d3be4a968165c6637f1a1590/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake