Abstract
Embodied AI robots have the potential to fundamentally improve the way human beings live and manufacture. Continued progress in the burgeoning field of using large language models to control robots depends critically on an efficient computing substrate. In particular, today's computing systems for embodied AI robots are designed purely based on the interest of algorithm developers, where robot actions are divided into a discrete frame-basis. Such an execution pipeline creates high latency and energy consumption. This paper proposes Corki, an algorithm-architecture co-design framework for real-time embodied AI robot control. Our idea is to decouple LLM inference, robotic control and data communication in the embodied AI robots compute pipeline. Instead of predicting action for one single frame, Corki predicts the trajectory for the near future to reduce the frequency of LLM inference. The algorithm is coupled with a hardware that accelerates transforming trajectory into actual torque signals used to control robots and an execution pipeline that parallels data communication with computation. Corki largely reduces LLM inference frequency by up to 8.0x, resulting in up to 3.6x speed up. The success rate improvement can be up to 17.3%. Code is provided for re-implementation. https://github.com/hyy0613/Corki
The Real-time Challenge in Embodied AI
While Large Language Models (LLMs) are brilliant at high-level reasoning, they are notoriously slow when applied to the physical world. In "Embodied AI"—where an LLM acts as the brain for a robot—this latency is a critical bottleneck.
Current systems typically follow a sequential, "vision-centric" execution model. In this pipeline, the robot captures an image, sends it to a cloud server, waits for the LLM to predict the next single action, and then executes it.
This creates a massive "frequency mismatch." Robotic control requires high-frequency signals (often 100Hz or more) for smooth and safe operation, but LLM inference and data communication can take hundreds of milliseconds. This results in jerky, high-latency movements that struggle to meet real-time constraints in dynamic environments.
The Corki Framework: Trajectory-Centric Control
To address these inefficiencies, the authors propose Corki, a framework that utilizes algorithm-architecture co-design to decouple high-level planning from low-level motor control.
Algorithmic Shift: Trajectory Prediction
Instead of asking the LLM for a single discrete step, Corki trains the model to predict a continuous trajectory for the near future using a cubic function ($at^3 + bt^2 + ct + d$). This "robotic-centric" approach allows the robot to follow a smooth path for several frames without needing a fresh LLM inference for every single step.
To maintain reactivity, Corki introduces Adaptive Trajectory Length. This mechanism uses a waypoint identification algorithm to decide if a trajectory should be terminated early—for instance, if the robot encounters a sudden obstacle or needs to make a sharp turn—ensuring the system remains responsive despite the longer planning horizon.
Architectural Innovation: Hardware Acceleration
The algorithmic shift is supported by a custom FPGA-based accelerator specifically for Task Space Computed Torque Control (TS-CTC)—the complex mathematics required to translate a planned path into actual electrical signals for robot motors.
- Data Reuse: The hardware minimizes redundant calculations by sharing intermediate results, such as joint poses and velocities, across different control blocks.
- Approximate Computing Enable (ACE) Unit: This optimization recognizes that control signals do not change drastically if the robot's joints are moving slowly. The ACE unit dynamically decides when to recompute complex matrices and when to reuse previous values, significantly saving power and time without sacrificing accuracy.
Key Results and Insights
The researchers evaluated Corki using the Calvin Dataset, a benchmark featuring 34 different tasks across nearly 23,000 demonstrations. By retraining the RoboFlamingo architecture to output trajectories, Corki achieved the following:
- System Speedup: Corki reduced LLM inference frequency by up to 5.1x, leading to an overall system speedup of up to 5.9x.
- Improved Accuracy: The success rate improved by up to 13.9% on seen tasks. Interestingly, predicting a smooth trajectory helped the model learn more effectively than predicting jerky, discrete steps.
- Hardware Efficiency: The custom accelerator reduced control latency by 29.0x compared to a standard CPU. The most aggressive variation (Corki-9) achieved a 9.2x reduction in energy consumption.
- Latency Hiding: By pipelining the system, Corki can capture and transmit new images to the server while the robot is still executing its current trajectory, effectively "hiding" communication latency behind computation.
Conclusion
Corki demonstrates that real-time Embodied AI is achievable through the strategic decoupling of planning and control. By shifting to a trajectory-based model and accelerating the underlying physics with specialized hardware, Corki provides a scalable path for deploying sophisticated LLMs in time-sensitive robotic applications.
Summary updated