Middle Code Prediction: Enhancing Code Generation for Uncommon Programming Languages in Robotics

Zixi Jia, Hongbin Gao, Hexiao LiACML 2024 · OpenReview

TL;DR

Researchers have developed a three-stage "Middle Code Prediction" (MCP) scheme that allows Large Language Models (LLMs) to generate code for uncommon robotic programming languages without manual API wrapping. By teaching LLMs to first predict a simplified "middle code" (akin to pseudo-code), the system achieves a 31% average accuracy boost and significantly better robustness against real-world "noisy" instructions.

What Problem Does This Paper Address?

While LLMs like GPT-4 or CodeLlama are wizards at Python or C++, they struggle with the "long tail" of programming—the thousands of domain-specific languages used in industrial robotics. Every robot brand often has its own proprietary, low-level language with very little training data available online.

To get around this, engineers usually have to manually "encapsulate" high-level APIs (e.g., creating a pick_up() function that hides 50 lines of low-level movement code). However, this is time-consuming, requires expert engineers, and often makes the robot less flexible because the LLM can only use the specific "blocks" the engineer provided. This paper tackles the challenge of generating long, complex, low-level code for these uncommon languages directly from natural language instructions.

Methods

The authors propose a three-stage framework called Middle Code Prediction (MCP). The core idea is inspired by how humans solve coding problems: we often write pseudo-code before diving into the actual syntax.

  1. Simplified Rule Interpretation: Humans provide a few high-quality examples (quadruples) consisting of the user instruction, the final low-level code, the simplification rules, and the resulting "middle code."
  2. Knowledge Extraction: The LLM uses these examples via few-shot prompting to learn how to translate between the abstract final code and the more readable middle code. It also generates an "extension script"—a simple mapping tool to turn middle code back into executable final code.
  3. Code Generation: When a new task arrives, the LLM predicts the middle code. Because middle code combines natural language reasoning with simplified syntax, it is much easier for the LLM to process. The extension script then automatically expands this into the final, brand-specific robotic code.

The "middle code" specifically relies on two modules: Sequential (ensuring the order of operations matches the instruction) and SubSummary (breaking down and summarizing sub-tasks).

Datasets

The researchers introduced a new benchmark:

  • Hospital Item Transport Dataset (HITD): A collection of natural language instructions and long sequences of low-level code for logistics robots. It features three difficulty levels (Task 1-3) based on the number of departments a robot must visit and whether "priority" tasks are involved.
  • Robopal Simulation: A desktop block-placement scenario used to test generalization.
  • Real-world Testing: The method was validated on a physical FANUC LR Mate 200iD/4S industrial robotic arm.

Key Insights and Ideas

  • Reducing Cognitive Load: The "aha moment" is that LLMs fail at uncommon languages not because they can't reason, but because the syntax is too abstract and the sequences are too long. Middle code acts as a "bridge" that aligns the robot's requirements with the LLM's natural language strengths.
  • Implicit Reasoning over Explicit Decomposition: Unlike other methods that explicitly force an LLM to "think step-by-step" in separate calls, MCP merges reasoning and code generation into a single flow, saving time and API costs.
  • Noise Robustness: One of the most interesting findings was that MCP-trained models are much better at handling "noise"—like synonyms, rephrased sentences, or irrelevant information—compared to models that were simply fine-tuned on final code. Because the middle code includes natural language reasoning, the model "understands" the intent rather than just memorizing code patterns.

Key Results

  • Accuracy Boost: MCP improved the mean accuracy of baseline models (including Llama 2 and GPT-3.5) by an average of 31%.
  • Stability: On the most difficult tasks (Task 3), standard methods often saw performance plummet to near zero, while MCP maintained stable, usable results.
  • Model Scaling: Even smaller models (7B parameters) became significantly more capable at handling long-code tasks when using the MCP framework, suggesting this could be a path toward deploying robotics LLMs on local, edge hardware.
  • Real-World Success: The system successfully drove a FANUC robotic arm to perform grasping tasks even in "No API" scenarios where no pre-written high-level functions existed.

Limitations and Future Work

The authors acknowledge two primary hurdles:

  1. Human-in-the-loop: Stage 1 still requires a "prompt engineer" to write the initial high-quality quadruples. While this is easier than writing a full API library, it still requires manual effort.
  2. Blind Reasoning: Currently, the system assumes the environment is static (object positions are pre-written in a file). The authors aim to integrate MCP with vision-language models (VLMs) in the future to allow robots to react to moving objects or dynamic obstacles in real-time.

Summary updated

Abstract

Generating executable code through natural language instructions to drive robotic movements is considered a crucial step towards achieving embodied intelligence. However, in the robotics domain, the scarcity of programming language data often necessitates manually encapsulating high-level APIs to enable Large Language Models(LLMs) to predict code correctly, which is time-consuming and incomplete. Therefore, this paper proposes a three-stage Middle Code Prediction(MCP) scheme, by injecting appropriate prompts at different stages, the LLMs can shift towards predicting middle code that it understands more easily. This middle code can then be converted into the final code through specific scripts, accomplishing the task of generating code in uncommon programming languages automatically and without the need for manually encapsulating high-level APIs. We tested our approach on Hospital Item Transport Dataset(HITD) and found that MCP could improve the mean accuracy of various baseline models to varying degrees, with an overall increase of 31%, while also enhancing the noise resistance of fine-tuned models. We conducted real-world experiments on industrial robotic arms, verifying the feasibility of MCP in scenarios with no API and partial API encapsulation. The method proposed in this paper provides a guideline for code generation in uncommon programming languages within the context of LLMs. Our experimental dataset is available at https://github.com/Ghbbbbb/MCP.
BibTeX
@inproceedings{jia2024middle,
  author = {Zixi Jia and Hongbin Gao and Hexiao Li},
  title = {{Middle Code Prediction: Enhancing Code Generation for Uncommon Programming Languages in Robotics}},
  booktitle = {ACML},
  year = {2024},
  url = {https://openreview.net/forum?id=GalIdp6HTx},
}
Figure