ActOR: Ambient AI-Based Agentic Orchestration for Humanoid Autonomy in Operating Rooms

Han Zhang, Sicheng Lu, Aleks Santari, Jeffrey K. Jopling, Masaru Ishii, Mathias UnberatharXiv 2026 · arXiv

TL;DR

ActOR (Ambient AI-Based Agentic Orchestration) enables humanoid robots to perform complex tasks in operating rooms by using wall-mounted cameras as a "global eye." By fusing ambient and egocentric views into a shared digital twin, a Large Language Model (LLM) agent can plan, execute, and recover from failures without the need for risky "de novo" exploration or task-specific training.

What Problem Does This Paper Address?

Operating rooms (ORs) are high-stakes, cluttered environments where efficiency is critical but staffing is often short. While humanoid robots are ideal for these spaces because they fit human-centric designs, they suffer from "tunnel vision." Relying solely on onboard cameras forces a robot to wander around (explore) to find objects, which is dangerous in a room full of sensitive medical equipment.

Furthermore, robotic execution is brittle. If a robot stops slightly too far from a table or misses a grasp, traditional "open-loop" systems often fail entirely. The authors address these challenges by asking: Can the instrumented OR itself become part of the robot’s perceptual infrastructure?

Methods

The ActOR framework consists of three main pillars:

  1. Shared Scene Representation: The system fuses data from a wall-mounted ambient camera and the robot’s head-mounted camera. This creates an object-level Digital Twin (DT)—a structured map containing object labels, 3D positions, and interaction states (e.g., "is this in the robot's hand?").
  2. Agentic Orchestration: A Large Language Model (specifically GPT-6.1 Sol) acts as the "brain." It receives the DT state as a JSON object and translates natural language instructions (like "Place all bottles in the biohazard bin") into a sequence of pre-programmed skills.
  3. Closed-Loop Feedback: Unlike systems that plan once and hope for the best, ActOR is "agentic." After every action, the DT is updated. If a skill fails—for instance, if an object is moved during navigation—the LLM receives the failure report and the updated scene state to decide whether to retry, change its approach, or replan.
Fig. 1: Overview of ActOR. (a) A humanoid executes a multi-object disposal task in a mock OR. (b) The shared scene representation integrates ambient and robot observations into an object-level DT and occupancy map. (c) An LLM agent uses the DT and robot skills for task planning, execution monitoring, and replanning.

Datasets and Setup

The researchers evaluated ActOR using both simulated and physical environments:

  • Simulation: They used TwinOR, a platform providing photorealistic digital twins of two real-world operating rooms (OR-A and OR-B). This allowed for controlled "ablation" studies to see exactly how much the ambient cameras and feedback loops helped.
  • Physical Mock OR: A real-world testbed equipped with an operating table, a mobile CT scanner, and a surgical robot.
  • Hardware: The physical experiments utilized a Unitree G1 humanoid equipped with Dex1-1 grippers and a ZED Mini head camera.
The experimental environments, including simulated ORs and the physical mock-up.

Key Insights and Ideas

  • Ambient AI as Infrastructure: The "aha moment" here is treating the room's sensors as a shared resource for the robot. This "eye in the sky" provides persistent context that egocentric vision simply cannot match in a cluttered space.
  • LLMs for Robustness: The study shows that LLMs are remarkably capable of handling "edge cases" in physical execution. By providing the LLM with a "failed skill note" (e.g., "grasp_missed"), the agent can reason through a recovery strategy just as a human might.
  • Eliminating Exploration: In the OR, the best exploration is no exploration. Ambient perception reduced the need for the robot to "look around" by nearly 90% when targets were initially out of sight.

Key Results

The quantitative gains from ActOR were substantial:

  • Exploration Burden: When targets were outside the robot's initial view, ambient perception reduced "Explore" calls from 1.05 down to 0.12 per trial.
  • Success under Pressure: In simulation, closed-loop feedback increased success rates from 46.7% to 83.3% in normal conditions. When the researchers intentionally sabotaged the robot (moving objects or inducing navigation errors), ActOR maintained a 51.7% success rate, while an open-loop baseline plummeted to just 8.3%.
  • Physical Validation: In the mock OR, the Unitree G1 successfully completed 14 out of 20 single-object disposal trials. It also demonstrated the ability to replan on the fly when a human moved the biohazard bin mid-task.
Simulation results showing task success rates and the reduction in exploration calls.

Limitations and Future Work

Despite the success, the authors note several hurdles for clinical deployment:

  • Privacy and Latency: The system currently relies on proprietary cloud-based LLMs. For a real hospital, these would need to be replaced with locally deployable, open-weight models to protect patient privacy.
  • Hardware Constraints: Humanoid robots still struggle with balance and motor overheating. The authors suggest that a wheeled base might be more stable for current OR tasks, though it would sacrifice the humanoid's ability to navigate narrow gaps.
  • Manipulation Complexity: The current skill library is limited to simple pick-and-place. Future work will need to address more complex interactions, such as opening cabinets or pushing heavy equipment carts.

Summary updated

Abstract

Operating rooms (ORs) increasingly contain ambient sensing infrastructure to enable AI-based optimization of workflows and utilization. We pursue an alternative path: using ambient AI to guide robot action and optimize utilization. Deploying humanoids in the OR without ambient AI is severely limited by cumbersome de novo orientation and planning due to partial onboard observations. We posit that this can be achieved by coupling an ambient AI-based agentic orchestrator with the humanoid robotic actor, providing shared perception and execution feedback for humanoid autonomy in the OR. ActOR fuses a wall-mounted ambient camera and the robot's head-mounted camera into a shared object-level digital twin (DT). A large language model agent plans over the DT using pre-programmed locomotion and manipulation skills without task-specific training. During execution, DT updates and execution outcomes inform retries, alternative approaches, or replanning. We evaluated ActOR in simulated and physical environments. Across three tasks in two simulated ORs reconstructed from real rooms, ambient perception reduced exploration from 1.05 to 0.12 Explore calls per trial when targets were initially outside the robot's view. Closed-loop feedback increased success from 46.7% to 83.3% under nominal execution and from 8.3% to 51.7% under injected navigation, object-relocation, and grasp perturbations. In a physical mock OR with a humanoid, ActOR completed 14 of 20 single-object disposal trials and demonstrated replanning after scene changes. Ambient AI-based observations provide scene state that improves robot action planning, increasing success and reducing exploration compared with robot-only observations. Feeding skill outcomes and DT updates back to the planner improves multi-step task success. Our results show the feasibility of humanoid task autonomy for multi-step disposal tasks in a mock OR.
BibTeX
@misc{zhang2026actor,
  author = {Han Zhang and Sicheng Lu and Aleks Santari and Jeffrey K. Jopling and Masaru Ishii and Mathias Unberath},
  title = {{ActOR: Ambient AI-Based Agentic Orchestration for Humanoid Autonomy in Operating Rooms}},
  year = {2026},
  eprint = {2611.04255},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04255},
}
Figure