TL;DR
Researchers have developed a "template-based prompting" architecture to orchestrate conversational agents that rely on multiple, heterogeneous information sources. By using a central Large Language Model (LLM) as an intelligent "glue," the system cleans up user input, routes questions to specialized expert modules, and polishes the final response into a natural-sounding dialogue.
What Problem Does This Paper Address?
Building a conversational agent that is both factually accurate and conversationally fluid is a major challenge.
- Rule-based systems (like Rasa or Watson) are reliable but often feel rigid and struggle with informal language or context.
- LLMs (like ChatGPT) are incredibly fluid but prone to "hallucinations" and lack access to specific, curated knowledge bases.
The authors address the coordination challenge: how do you build a system that uses specialized "answerer" modules (which might be based on ontologies, rules, or specific datasets) while maintaining the smooth user experience of an LLM? Specifically, they tackle four hurdles: Sensitivity (handling typos/slang), State (remembering context across turns), Selection (choosing the right expert for the job), and Output (turning blunt, factual data into a nice sentence).
Methods
The core of their solution is a client-server architecture where a central LLM (GPT-3.5) acts as an orchestrator rather than the primary knowledge source. The process follows a three-step pipeline for every user query:
- Correction: The LLM receives the raw user message and the dialogue history. It rewrites the query into a single, self-contained, grammatically correct sentence. This resolves "anaphora" (e.g., changing "Where is it?" to "Where is the Blue Amazon located?").
- Redirection: The LLM is given descriptions of available specialized "answerers" and must decide which one is best suited to answer the corrected query. If none fit, it flags the message as out-of-scope.
- Completion: Once a specialized module provides a raw answer (which might just be a date or a single word), the LLM wraps that answer back into a complete, polite, and contextualized sentence for the user.
This approach is "loosely coupled," meaning new expert modules can be added easily by simply updating the LLM's redirection prompt with a new description.
Datasets
The authors evaluated their orchestrator using a custom dataset tailored to their use case, BLAB (Blue Amazon Brain), which focuses on the Brazilian maritime territory.
- Test Set: 50 manually selected question-answer pairs.
- 15 pairs for DEEPAGÉ (a specialized Portuguese QA system).
- 20 pairs for a Watson Assistant bot.
- 15 pairs that were out-of-scope (unsupported domains or malformed questions).
- Ground Truth: Each pair was manually labeled with the correct "answerer" ID.
Key Insights and Ideas
- LLM as Interpreter, Not Encyclopedia: Instead of asking the LLM for facts, they use its linguistic strength to "translate" between messy human speech and the rigid input requirements of specialized databases.
- The "ONLY" Trick: The researchers found that to get reliable programmatic output from GPT-3.5 (like a single digit for routing), they had to use capitalized instructions like "Send ONLY the number, without text" to prevent the model from adding conversational filler that breaks parsers.
- Contextual Independence: By forcing the "Correction" step to produce a self-contained sentence, they allow specialized modules—which usually have no "memory"—to function perfectly within a long conversation.
- Preserving Accuracy: The "Completion" step is explicitly instructed not to interfere with the factual content of the answerer's output, acknowledging that the specialized module is the "source of truth," not the LLM.
Key Results
The evaluation focused on the Redirection (routing) accuracy:
- Domain Accuracy: For well-formed questions within the supported domain, the system achieved a 90% success rate in sending the question to the correct expert module.
- Out-of-Scope Challenges: The system struggled to identify when a question was outside its expertise, correctly identifying out-of-scope queries only 27% of the time.
- Qualitative Improvement: The authors demonstrated that the "Correction" step allowed the system to answer questions correctly even when the user included typos (e.g., "patrimonio" vs "patrimônio") that would normally cause the specialized modules to fail.
Limitations and Future Work
- Out-of-Scope Detection: The current prompt-based redirection is not yet robust enough to filter out irrelevant or malformed questions reliably.
- Small Talk: The current architecture is strictly task-oriented. The authors want to integrate a way for the LLM to handle "small talk" without diverting the user away from the specialized knowledge modules.
- Latency: While not explicitly measured in the paper, a three-call LLM pipeline (Correction -> Redirection -> Completion) typically introduces more latency than a single-call system.
- User Testing: The authors plan to conduct extensive user testing to evaluate the "Output" challenge—specifically, how much the "Completion" step actually improves the perceived user experience.
Summary updated