Introduction
Building conversational agents that are both flexible and reliable remains a significant challenge in artificial intelligence. While Large Language Models (LLMs) have revolutionized natural language understanding, they often struggle with strict adherence to business policies, hallucinate information, and fail to integrate diverse knowledge sources. This paper introduces Genie, a programmable framework designed to bring order to LLM-based agents. By utilizing a declarative specification called Genie Worksheets, the framework separates conversation logic from language generation, significantly improving goal completion rates in complex, real-world scenarios.
The Challenge of LLM-Based Agents
Pure LLM-based agents typically operate in a "wild west" environment where they lack structured control over dialogue flow. They often fail to follow complex conditional logic and struggle to synthesize information from multiple sources, such as structured databases and unstructured text manuals. Conversely, traditional task-oriented systems rely on rigid dialogue trees or slot-filling mechanisms. While reliable, these systems are brittle; they cannot easily handle unexpected user questions or out-of-order information. Genie aims to provide a middle ground: the flexibility of an LLM combined with the deterministic control of a traditional program.
The Genie Framework
The Genie architecture is built on the principle of separating the what (logic and state) from the how (natural language expression). The system is divided into four primary components that work in concert to manage the user interaction.
- Genie Worksheet: A high-level declarative blueprint defined by the developer.
- Genie Parser: An LLM that translates natural language into formal state updates (Python-like code).
- Genie Runtime: A non-LLM, algorithmic "brain" that manages dialogue state, executes queries (using the SUQL language), and enforces policies.
- Genie Response Generator: An LLM that converts formal instructions from the Runtime back into human-sounding responses.
By delegating logic to a deterministic algorithm and language to an LLM, Genie prevents agents from wandering off-script or generating inaccurate information.
Declarative Programming with Genie Worksheets
The core innovation of the framework is the Genie Worksheet. Rather than mapping every possible dialogue path, developers define a schema of the task. This includes the required information (fields), the rules governing the task (predicates), and the resulting actions once data is collected.
This declarative approach allows for several key advantages:
- State Compression: Instead of processing a messy chat history, the LLM receives a clean, structured representation of the current Worksheet, reducing hallucinations.
- Compositionality: The system can handle nested requests, such as searching for a restaurant's rating before initiating a reservation API call.
- Flexibility: Because the system focuses on reaching a defined state rather than following a flowchart, it can naturally handle users who change their minds or provide information out of sequence.
Evaluation and Results
Genie was evaluated using the StarV2 benchmark—a complex dataset for logic-heavy dialogues—and through user studies involving three real-world applications: restaurant reservations (using Yelp data), university ticket submission, and course enrollment.
The results demonstrate a substantial leap in reliability. On the StarV2 benchmark, Genie outperformed previous state-of-the-art methods by up to 20.5%. In user studies, a standard GPT-4 Turbo agent using traditional function calling completed tasks only 21.8% of the time. In contrast, the Genie-powered version achieved an 82.8% completion rate. Furthermore, the framework proved efficient enough that smaller models, like GPT-4o-mini, could outperform larger base models when equipped with Genie.
Conclusion and Future Work
Genie represents a major step toward reliable, programmable conversational agents. While it significantly reduces hallucinations and logic errors, it remains dependent on the LLM's ability to accurately parse natural language into formal code. Future research will focus on scaling the system to handle larger libraries of worksheets through few-shot retrieval and further reducing the developer effort required to create these declarative specifications.
Summary updated