SayComply: Grounding Field Robotic Tasks in Operational Compliance through Retrieval-Based Language Models

Muhammad Fadhil Ginting, Dong-Ki Kim, Sung-Kyun Kim, Bandi Jai Krishna, Mykel J. Kochenderfer, Shayegan Omidshafiei, Ali-akbar Agha-mohammadi

Abstract

SayComply is a novel framework designed to ground robotic task planning in domain-specific operational manuals using a hierarchical retrieval-augmented generation (RAG) approach. By structuring technical documentation into a three-level hierarchy, the system enables autonomous robots to execute user instructions while strictly adhering to safety protocols and site-specific procedures not present in general-purpose Large Language Model (LLM) training data. Our evaluation demonstrates that SayComply achieves a 91.4% success rate in both compliance and task completion, significantly outperforming standard RAG baselines in complex industrial scenarios.

1. Introduction

While Large Language Models (LLMs) demonstrate impressive general reasoning capabilities, they often lack the "proprietary common sense" required for specialized industrial environments. For instance, a robot tasked with inspecting a refinery must not only navigate the environment but also adhere to specific safety protocols, interpret specialized gauges correctly, and consider equipment maintenance history. Existing methods typically rely on general knowledge or require expensive fine-tuning for each new site. Furthermore, standard RAG techniques often suffer from "context stuffing" or the "lost in the middle" phenomenon when processing massive technical manuals. SayComply addresses these challenges by ensuring operational compliance through grounded, hierarchical technical documentation.

2. Methodology

The SayComply framework consists of two primary components: a Hierarchical Context Database and a specialized Compliant Task Planner.

Overview of the SayComply architecture, integrating hierarchical RAG with a compliant task planner.

2.1 Hierarchical RAG

To manage complex documentation, SayComply organizes information into a three-level hierarchy:

  • Level 1 (Observations): Dynamic data, including past inspection logs and current robot states. To handle tabular data where standard vector embeddings often fail, the system uses an LLM to reason over metadata and log titles.
  • Level 2 (Site-specific Details): High-level instructions or "cheat sheets" provided by site experts.
  • Level 3 (High-level Manuals): Extensive technical documents, such as safety manuals.

2.2 Tree-Based Retrieval and Planning

The system employs a "Tree-RAG" approach where user queries are first matched against Level 2 summaries. These summaries serve as pointers to relevant detailed sections in Level 3, reducing irrelevant noise. The Compliant Task Planner then treats task generation as a multi-objective optimization problem, using a weight constant ($\beta \gg 1$) to prioritize safety and protocol compliance over simple task completion. If a request violates established manuals, the planner is designed to refuse the task and provide feedback.

3. Evaluation

3.1 Dataset and Scenarios

We developed a new evaluation suite comprising 62 distinct sources, including industrial inspection manuals and robot embodiment safety documents. The benchmark includes 70 diverse queries across three tracks: Industrial Inspection (Oil & Gas), Office Operations, and Embodiment-aware operations.

3.2 Results

SayComply was validated in NVIDIA’s Isaac Sim and on a physical Boston Dynamics Spot robot. Key results include:

  • Performance: A 91.4% success rate in compliance and task completion, compared to 70% for standard RAG.
  • Retrieval Accuracy: The hierarchical approach achieved 92.9% accuracy, significantly higher than the 65.7% baseline.
  • Hardware Validation: On the Spot robot, the system successfully utilized site-specific context to navigate and inspect AC control panels correctly.

4. Discussion and Conclusion

SayComply demonstrates that mimicking human learning—moving from quick-start guides (Level 2) to deep manuals (Level 3)—is more efficient for robotic retrieval than flat vector embeddings. Additionally, prioritizing refusal in high-stakes environments ensures safety. Future work will explore role-based authority levels for overriding protocols and optimizing the system for onboard "edge" deployment without cloud connectivity.

Summary updated

BibTeX
@inproceedings{ginting2025saycomply,
  author = {Muhammad Fadhil Ginting and Dong-Ki Kim and Sung-Kyun Kim and Bandi Jai Krishna and Mykel J. Kochenderfer and Shayegan Omidshafiei and Ali-akbar Agha-mohammadi},
  title = {{SayComply: Grounding Field Robotic Tasks in Operational Compliance through Retrieval-Based Language Models}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2025},
  doi = {10.1109/icra55743.2025.11128684},
}
Figure