TL;DR
This paper introduces Semantic In-Context Learning (S-ICL), a novel approach for conversational agents that combines semantic search with large language models (LLMs) like ChatGPT. S-ICL addresses the challenge of enabling LLMs to leverage private, domain-specific knowledge without costly fine-tuning, achieving competitive results in a recent NLP competition.
What Problem Does This Paper Address?
Modern conversational agents, powered by LLMs, are incredibly capable but face two major hurdles. First, they often lack access to an organization's private, internal knowledge (like FAQs or policies), limiting their ability to provide accurate, context-specific answers. Second, fine-tuning LLMs on this private data is expensive, time-consuming, and sometimes not even an option for proprietary models like ChatGPT. This means agents trained on public data might struggle with current events or specific company information. The paper aims to bridge this gap by allowing LLMs to effectively utilize private data for more informed responses.
Methods
The proposed Semantic In-Context Learning (S-ICL) architecture comprises several key components:
- Data Pre-processor: This module takes conversation logs (utterances between a student and teacher) in JSON format and transforms them into a concatenated text format for semantic search and a "Sample" format for the LLM.
- Embedder: A pre-trained transformer encoder model (specifically "multi-qa-mpnet-base-dot-v1") converts the concatenated text into dense vector embeddings.
- Retriever: This component uses cosine similarity to find the most relevant past conversation samples from a training set based on the embedding of the current inquiry. It retrieves the top N similar records.
- Prompt Builder: This module constructs the prompt for the LLM. It includes a command instructing the LLM, the retrieved "Sample(s)" from the training set, and the current "Inquiry" (context and user query). The authors explore various prompt building approaches.
- Answer Generator: A large language model (ChatGPT with gpt-3.5-turbo engine) receives the constructed prompt and generates a response. The temperature is set to zero for reproducibility.
The core innovation lies in using a semantic search engine (the Embedder and Retriever) to dynamically fetch relevant examples from private data, which are then fed into the LLM's prompt as context, enabling in-context learning without explicit fine-tuning.
Datasets
The primary dataset used is the TSCC dataset, which consists of conversations between a student and a teacher, provided by Caines et al. (2020). The authors split this dataset into customized training and testing sets. They also mention participating in the BEA 2023 shared task, which likely involved their own evaluation sets derived from the TSCC dataset.
Key Insights and Ideas
- Semantic Search as a Knowledge Augmenter: The paper demonstrates that semantic search can effectively act as a bridge, bringing private, domain-specific knowledge to LLMs without the need for fine-tuning.
- Prompt Engineering Matters: The authors found that the way the prompt command is phrased significantly impacts the LLM's response length and conciseness. Specifically, framing the request as "Find the appropriate answer by the teacher from sample X..." led to more concise outputs.
- Fixed Samples Can Be Effective: Surprisingly, using a single, curated "fixed sample" for all inquiries (the third approach) yielded the best BERTScore in terms of F1-score. This suggests that for certain tasks, a well-chosen demonstration can be robust.
- Human Preference is Key: While one approach might score highest on automated metrics (like BERTScore), another might be preferred by human evaluators (as seen with the fifth approach being more "reasonable and preferable"). This highlights the importance of human evaluation in conversational AI.
Key Results
The proposed S-ICL approach participated in the BEA 2023 shared task and achieved fourth place in both the development and evaluation phases.
In their internal experiments on a custom test set:
- Approach 3 (fixed sample) achieved the best BERTScore F1-score.
- Approach 2 achieved the best DialogRPT scores.
- Approach 5 was found to produce more "reasonable and preferable" answers from a human perspective.
Limitations and Future Work
- Prompt Engineering Complexity: The authors acknowledge that prompt engineering is crucial and can be complex. They suggest that further efforts in this area could significantly improve the architecture's performance.
- Response Length: They observed that ChatGPT tends to generate longer responses than ground truths, though prompt phrasing helped mitigate this.
- Future Work: The paper suggests that the architecture is flexible and can be adapted to different domains (e.g., news media, customer service). Further exploration of prompt building approaches is also a promising direction. They also note that their model could not proceed to human evaluation in the BEA task due to an emergency, implying that human evaluation would be a valuable next step.
Summary updated