TL;DR
ActOR (Ambient AI-Based Agentic Orchestration) enables humanoid robots to perform complex tasks in operating rooms by using wall-mounted cameras as a "global eye." By fusing ambient and egocentric views into a shared digital twin, a Large Language Model (LLM) agent can plan, execute, and recover from failures without the need for risky "de novo" exploration or task-specific training.
What Problem Does This Paper Address?
Operating rooms (ORs) are high-stakes, cluttered environments where efficiency is critical but staffing is often short. While humanoid robots are ideal for these spaces because they fit human-centric designs, they suffer from "tunnel vision." Relying solely on onboard cameras forces a robot to wander around (explore) to find objects, which is dangerous in a room full of sensitive medical equipment.
Furthermore, robotic execution is brittle. If a robot stops slightly too far from a table or misses a grasp, traditional "open-loop" systems often fail entirely. The authors address these challenges by asking: Can the instrumented OR itself become part of the robot’s perceptual infrastructure?
Methods
The ActOR framework consists of three main pillars:
- Shared Scene Representation: The system fuses data from a wall-mounted ambient camera and the robot’s head-mounted camera. This creates an object-level Digital Twin (DT)—a structured map containing object labels, 3D positions, and interaction states (e.g., "is this in the robot's hand?").
- Agentic Orchestration: A Large Language Model (specifically GPT-6.1 Sol) acts as the "brain." It receives the DT state as a JSON object and translates natural language instructions (like "Place all bottles in the biohazard bin") into a sequence of pre-programmed skills.
- Closed-Loop Feedback: Unlike systems that plan once and hope for the best, ActOR is "agentic." After every action, the DT is updated. If a skill fails—for instance, if an object is moved during navigation—the LLM receives the failure report and the updated scene state to decide whether to retry, change its approach, or replan.
Datasets and Setup
The researchers evaluated ActOR using both simulated and physical environments:
- Simulation: They used TwinOR, a platform providing photorealistic digital twins of two real-world operating rooms (OR-A and OR-B). This allowed for controlled "ablation" studies to see exactly how much the ambient cameras and feedback loops helped.
- Physical Mock OR: A real-world testbed equipped with an operating table, a mobile CT scanner, and a surgical robot.
- Hardware: The physical experiments utilized a Unitree G1 humanoid equipped with Dex1-1 grippers and a ZED Mini head camera.
Key Insights and Ideas
- Ambient AI as Infrastructure: The "aha moment" here is treating the room's sensors as a shared resource for the robot. This "eye in the sky" provides persistent context that egocentric vision simply cannot match in a cluttered space.
- LLMs for Robustness: The study shows that LLMs are remarkably capable of handling "edge cases" in physical execution. By providing the LLM with a "failed skill note" (e.g., "grasp_missed"), the agent can reason through a recovery strategy just as a human might.
- Eliminating Exploration: In the OR, the best exploration is no exploration. Ambient perception reduced the need for the robot to "look around" by nearly 90% when targets were initially out of sight.
Key Results
The quantitative gains from ActOR were substantial:
- Exploration Burden: When targets were outside the robot's initial view, ambient perception reduced "Explore" calls from 1.05 down to 0.12 per trial.
- Success under Pressure: In simulation, closed-loop feedback increased success rates from 46.7% to 83.3% in normal conditions. When the researchers intentionally sabotaged the robot (moving objects or inducing navigation errors), ActOR maintained a 51.7% success rate, while an open-loop baseline plummeted to just 8.3%.
- Physical Validation: In the mock OR, the Unitree G1 successfully completed 14 out of 20 single-object disposal trials. It also demonstrated the ability to replan on the fly when a human moved the biohazard bin mid-task.
Limitations and Future Work
Despite the success, the authors note several hurdles for clinical deployment:
- Privacy and Latency: The system currently relies on proprietary cloud-based LLMs. For a real hospital, these would need to be replaced with locally deployable, open-weight models to protect patient privacy.
- Hardware Constraints: Humanoid robots still struggle with balance and motor overheating. The authors suggest that a wheeled base might be more stable for current OR tasks, though it would sacrifice the humanoid's ability to navigate narrow gaps.
- Manipulation Complexity: The current skill library is limited to simple pick-and-place. Future work will need to address more complex interactions, such as opening cabinets or pushing heavy equipment carts.
Summary updated