TL;DR
Large Language Models (LLMs) are increasingly used in autonomous systems, but their "hallucinations" pose significant risks. This paper argues that instead of just trying to make LLMs perfect, we should treat them as "imperfect components"—much like a noisy sensor—and use classical engineering methods like bandits and Kalman filters to dynamically adjust how much a system relies on them.
What Problem Does This Paper Address?
As LLMs move from chatbots to robotic controllers and planners, their fallibility becomes a safety and efficiency bottleneck. Most current research focuses on improving the LLM itself through better prompting (context updates) or fine-tuning (parameter updates).
However, the authors identify a gap: how should the system surrounding the LLM manage its output? If an LLM provides a plan that is 60% likely to be correct, when should the robot execute it, when should it ask for a human check, and when should it switch to a traditional symbolic planner? This paper provides a framework for adapting reliance—changing how a system uses an LLM based on accumulated evidence of its performance.
Methods
The paper synthesizes decades of classical control theory, decision theory, and online learning to create a roadmap for LLM integration. The core idea is the Reliance Update, contrasted with traditional methods in the figure below:
The authors break down the problem into four steps:
- Identify the Functional Role: Is the LLM a planner, a reward function architect, or a low-level controller?
- Define Reliance Dimensions: How can we vary trust? Options include Assignment (choosing between sources), Oversight (deciding when to verify), and Latitude (restricting the LLM's range of choices).
- Estimate Reliability: Use validators, simulators, or physical measurements to calculate error rates (e.g., false-acceptance rates for an LLM acting as a judge).
- Apply Classical Algorithms: Map reliance dimensions to established methods:
- Contextual Bandits: To learn which LLM (or traditional tool) works best in a specific context.
- Kalman Filters: To fuse LLM predictions with physical sensor data, weighting the LLM less when its uncertainty is high.
- Value of Information (VoI): To decide if the cost of verifying an LLM's plan is worth the potential risk reduction.
Datasets
As a review paper, it does not introduce a new dataset but analyzes results from several benchmarks and systems:
- Blocksworld: A classic planning domain used to evaluate GPT-4's ability to rearrange blocks.
- Sudoku: Used to test oversight mechanisms for LLM-based solvers.
- COCO-TAMP: A robotic search system combining LLM knowledge with Bayesian state estimation.
- BORA & CALIPER: Bayesian optimization frameworks that use LLMs to guide searches in high-dimensional spaces.
Key Insights and Ideas
- LLMs as Noisy Sensors: The most powerful conceptual shift is treating LLM output as "evidence" rather than "truth." This allows engineers to use 50+ years of robust control theory to handle "hallucinations" as mere signal noise.
- The Cost of Oversight: Verifying an LLM isn't free. The paper notes that using an LLM to check another LLM's work can be counterproductive if the "checker" is also fallible. In one study, a GPT-4 evaluator accepted 38 out of 45 invalid plans, showing that "self-correction" is often an illusion.
- Propensity Weighting: If a system stops using an LLM because it's performing poorly, it will never know if the LLM improves (e.g., after a server-side update). Systems must occasionally "sample" the LLM's performance to keep their reliability estimates fresh.
- Safety vs. Utility: For critical requirements, statistical reliability isn't enough. The authors advocate for independently verified mechanisms—like symbolic validators or Control Barrier Functions—that act as a hard "shield" regardless of what the LLM proposes.
Key Results
- Efficiency Gains: In the COCO-TAMP system, using LLM-derived priors about where objects are typically found reduced planning and execution time by 62.7%.
- The Failure of Direct Reliance: While LLM priors helped, a variant that let the LLM update location beliefs directly failed in 3 out of 5 trials.
- Superior Optimization: The BORA system, which only asks an LLM for guidance when its internal optimizer stalls, achieved values up to 67% higher than systems relying on the LLM alone.
- Reliability Bounds: Using conformal prediction, systems like KnowNo can bound the risk of task failure to a user-specified probability by knowing exactly when to stop and ask a human for help.
Limitations and Future Work
- Correlated Errors: A major risk is that different LLMs (or an LLM and its verifier) might share training data, leading them to make the same mistakes. This breaks the assumption of independence required by many classical fusion methods.
- Non-Stationarity: LLMs are "moving targets." Performance can shift due to hidden model updates by providers. Developing methods to detect these shifts quickly is a priority.
- Domain Knowledge Integration: The authors highlight an open question: how can we best combine "black-box" statistical performance data with "white-box" symbolic domain knowledge to decide when to trust the model?
Summary updated