LLM-Mediated Domain-Specific Voice Agents: The Case of TextileBot

Shu Zhong, Elia Gatti, James Hardwick, Miriam Ribul, Youngjun Cho, Marianna ObristarXiv 2024 · DOI · arXiv

Introduction

Developing conversational agents (CAs) for specialized domains, such as textile circularity and sustainable fashion, presents significant challenges. Traditional approaches rely on extensive, hand-annotated datasets that are costly to produce. While Large Language Models (LLMs) provide a powerful alternative, they are prone to hallucinations—confidently stating factual errors—and often lack the contextual memory required for effective multi-turn voice interactions.

This paper introduces TextileBot, a voice-based agent designed to bridge the gap between general-purpose LLMs and domain-specific expertise. By employing a novel "Taxonomy-based Knowledge Structure Chain," the system grounds the LLM in verified domain knowledge without the need for expensive fine-tuning.

System Architecture and Methodology

TextileBot utilizes a multi-model pipeline to manage the complexities of voice interaction. The architecture transitions from traditional modular pipelines toward a more integrated LLM-based reasoning engine.

Comparison of traditional and LLM-based voice agent architectures.

The system is composed of four primary modules:

  1. Speech Recognition (ASR): OpenAI’s Whisper model for robust audio-to-text conversion.
  2. Reasoning Engine (LLM): GPT-3.5 (text-davinci-003) serves as the central processor.
  3. Speech Synthesis (TTS): Google Text-to-Speech for generating audio responses.
  4. Dialogue Memory: A custom module that summarizes interaction history to maintain context across conversational turns.

Taxonomy-based Knowledge Grounding

To mitigate hallucinations, the researchers developed a three-phase framework that translates expert domain knowledge into a format the LLM can reliably process. Using TextileNet, a comprehensive material taxonomy, the system injects a hierarchical "ground truth" map into the model's prompts. This ensures that relationships (e.g., identifying viscose as a bio-derived fiber rather than a synthetic one) are maintained accurately.

The process of transforming expert domain knowledge into structured chain prompts.

Experimental Design: The Role of Persona

The study explored how different prompt engineering strategies affect user engagement. Three distinct personas were tested:

  • Vanilla: The baseline LLM without specific domain instructions.
  • Expert: A technical persona focused strictly on precise textile circularity data.
  • Assistant: A social persona that balances technical knowledge with friendly "chit-chat."

A user study involving 30 participants generated a corpus of 93 dialogues and 1,272 conversational turns to evaluate these personas.

Key Findings

The "Expert Trap" and Persona Preference

Results indicated that while the "Expert" persona was technically accurate, users found it overly verbose and "teacher-like," leading to lower engagement. In contrast, the Assistant persona was the most successful, facilitating an average of 17.6 conversational turns compared to 11.0 for the Expert. Users were more likely to forgive minor errors when the agent employed social protocols, such as greetings or humor.

Fault Tolerance and Error Correction

A significant technical insight was the "auto-correct" effect provided by domain-specific grounding. The taxonomy-based approach successfully mitigated 62% of ASR errors. For example, if the ASR misheard "textile circularity" as "Texas secularity," the grounded LLM used the surrounding context to correctly interpret the user's intent and maintain the conversation's relevance.

Educational Impact

The system proved highly effective as an educational tool. Despite 70% of participants having no prior knowledge of the subject, 90% were able to accurately define textile circularity after their interaction with TextileBot.

Conclusion and Future Work

TextileBot demonstrates that structured prompt engineering can effectively transform general LLMs into domain experts. The study highlights that for specialized voice agents, the "persona" is as critical as factual precision for sustaining user interest. Future research will focus on reducing hardware latency for real-time retail applications and integrating Retrieval-Augmented Generation (RAG) to further enhance the system's factual reliability in data-intensive domains.

Summary updated

Abstract

Developing domain-specific conversational agents (CAs) has been challenged by the need for extensive domain-focused data. Recent advancements in Large Language Models (LLMs) make them a viable option as a knowledge backbone. LLMs behaviour can be enhanced through prompting, instructing them to perform downstream tasks in a zero-shot fashion (i.e. without training). To this end, we incorporated structural knowledge into prompts and used prompted LLMs to prototyping domain-specific CAs. We demonstrate a case study in a specific domain-textile circularity - TextileBot, we present the design, development, and evaluation of the TextileBot. Specially, we conducted an in-person user study (N=30) with Free Chat and Information-Gathering tasks with TextileBots to gather insights from the interaction. We analyse the human-agent interactions, combining quantitative and qualitative methods. Our results suggest that participants engaged in multi-turn conversations, and their perceptions of the three variation agents and respective interactions varied demonstrating the effectiveness of our prompt-based LLM approach. We discuss the dynamics of these interactions and their implications for designing future voice-based CAs.
BibTeX
@misc{zhong2024llm,
  author = {Shu Zhong and Elia Gatti and James Hardwick and Miriam Ribul and Youngjun Cho and Marianna Obrist},
  title = {{LLM-Mediated Domain-Specific Voice Agents: The Case of TextileBot}},
  year = {2024},
  eprint = {2406.10590},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.10590},
}
Figure