Coordination within Conversational Agents with Multiple Sources

Vinícius Bitencourt Matos, Ricardo S. Grava, Rodrigo Tavares, Marcos M. José, Paulo Pirozelli, Anarosa A. F. Brandão, Sarajane Marques Peres, Fábio Gagliardi Cozman2023 · DOI

TL;DR

Researchers have developed a "template-based prompting" architecture to orchestrate conversational agents that rely on multiple, heterogeneous information sources. By using a central Large Language Model (LLM) as an intelligent "glue," the system cleans up user input, routes questions to specialized expert modules, and polishes the final response into a natural-sounding dialogue.

What Problem Does This Paper Address?

Building a conversational agent that is both factually accurate and conversationally fluid is a major challenge.

  • Rule-based systems (like Rasa or Watson) are reliable but often feel rigid and struggle with informal language or context.
  • LLMs (like ChatGPT) are incredibly fluid but prone to "hallucinations" and lack access to specific, curated knowledge bases.

The authors address the coordination challenge: how do you build a system that uses specialized "answerer" modules (which might be based on ontologies, rules, or specific datasets) while maintaining the smooth user experience of an LLM? Specifically, they tackle four hurdles: Sensitivity (handling typos/slang), State (remembering context across turns), Selection (choosing the right expert for the job), and Output (turning blunt, factual data into a nice sentence).

Methods

The core of their solution is a client-server architecture where a central LLM (GPT-3.5) acts as an orchestrator rather than the primary knowledge source. The process follows a three-step pipeline for every user query:

  1. Correction: The LLM receives the raw user message and the dialogue history. It rewrites the query into a single, self-contained, grammatically correct sentence. This resolves "anaphora" (e.g., changing "Where is it?" to "Where is the Blue Amazon located?").
  2. Redirection: The LLM is given descriptions of available specialized "answerers" and must decide which one is best suited to answer the corrected query. If none fit, it flags the message as out-of-scope.
  3. Completion: Once a specialized module provides a raw answer (which might just be a date or a single word), the LLM wraps that answer back into a complete, polite, and contextualized sentence for the user.

This approach is "loosely coupled," meaning new expert modules can be added easily by simply updating the LLM's redirection prompt with a new description.

Datasets

The authors evaluated their orchestrator using a custom dataset tailored to their use case, BLAB (Blue Amazon Brain), which focuses on the Brazilian maritime territory.

  • Test Set: 50 manually selected question-answer pairs.
  • 15 pairs for DEEPAGÉ (a specialized Portuguese QA system).
  • 20 pairs for a Watson Assistant bot.
  • 15 pairs that were out-of-scope (unsupported domains or malformed questions).
  • Ground Truth: Each pair was manually labeled with the correct "answerer" ID.

Key Insights and Ideas

  • LLM as Interpreter, Not Encyclopedia: Instead of asking the LLM for facts, they use its linguistic strength to "translate" between messy human speech and the rigid input requirements of specialized databases.
  • The "ONLY" Trick: The researchers found that to get reliable programmatic output from GPT-3.5 (like a single digit for routing), they had to use capitalized instructions like "Send ONLY the number, without text" to prevent the model from adding conversational filler that breaks parsers.
  • Contextual Independence: By forcing the "Correction" step to produce a self-contained sentence, they allow specialized modules—which usually have no "memory"—to function perfectly within a long conversation.
  • Preserving Accuracy: The "Completion" step is explicitly instructed not to interfere with the factual content of the answerer's output, acknowledging that the specialized module is the "source of truth," not the LLM.

Key Results

The evaluation focused on the Redirection (routing) accuracy:

  • Domain Accuracy: For well-formed questions within the supported domain, the system achieved a 90% success rate in sending the question to the correct expert module.
  • Out-of-Scope Challenges: The system struggled to identify when a question was outside its expertise, correctly identifying out-of-scope queries only 27% of the time.
  • Qualitative Improvement: The authors demonstrated that the "Correction" step allowed the system to answer questions correctly even when the user included typos (e.g., "patrimonio" vs "patrimônio") that would normally cause the specialized modules to fail.

Limitations and Future Work

  • Out-of-Scope Detection: The current prompt-based redirection is not yet robust enough to filter out irrelevant or malformed questions reliably.
  • Small Talk: The current architecture is strictly task-oriented. The authors want to integrate a way for the LLM to handle "small talk" without diverting the user away from the specialized knowledge modules.
  • Latency: While not explicitly measured in the paper, a three-call LLM pipeline (Correction -> Redirection -> Completion) typically introduces more latency than a single-call system.
  • User Testing: The authors plan to conduct extensive user testing to evaluate the "Output" challenge—specifically, how much the "Completion" step actually improves the perceived user experience.

Summary updated

Abstract

Conversational agents can now operate with language models, rules, ontologies and varied other sources to provide smooth dialogue. However, the coordination of multiple sources in conversational agents is a challenge. We present a mechanism to effectively orchestrate multiple sources in a conversational agent, by relying on a client-server approach with an associated prompt generation module that deals with heterogeneous domain-oriented modules. As a detailed use case, we describe the architecture of a chatbot specialised in topics related to the Brazilian coast, and we study the benefits of our approach.
BibTeX
@misc{matos2023coordination,
  author = {Vinícius Bitencourt Matos and Ricardo S. Grava and Rodrigo Tavares and Marcos M. José and Paulo Pirozelli and Anarosa A. F. Brandão and Sarajane Marques Peres and Fábio Gagliardi Cozman},
  title = {{Coordination within Conversational Agents with Multiple Sources}},
  year = {2023},
  doi = {10.5753/eniac.2023.234533},
}
Figure