TL;DR
This paper introduces Effervesce, a novel conversational AI chatbot designed to integrate seamlessly into group discussions within digital workspaces like Slack. By fine-tuning a Large Language Model (LLM) on multi-actor conversation data, Effervesce aims to enhance team communication, moving beyond traditional one-to-one AI interactions.
What Problem Does This Paper Address?
In today's increasingly digital and often distributed work environments, effective communication within teams is paramount. While online collaboration software (OCS) facilitates this, facilitating natural, dynamic, and multi-actor conversations remains a challenge. Existing conversational AI tools are largely designed for one-on-one interactions, failing to capture the nuances of group dynamics, multiple speakers, and evolving conversational contexts. This paper tackles the problem of creating an AI that can actively participate in and enhance these complex group conversations, acting as a true team member.
Methods
The researchers developed Effervesce by first evaluating several open-source Large Language Models (LLMs) on a custom dataset of group conversations. They then selected the best-performing model and fine-tuned it using a Parameter-Efficient Fine-Tuning (PEFT) technique called QLoRA. This method allows for efficient fine-tuning by updating only a small fraction of the model's parameters, reducing computational costs and environmental impact while aiming to preserve the model's overall performance. The fine-tuned model was then integrated into a Slack environment via the Slack API, allowing it to receive messages, process conversation history, and generate real-time responses.
Datasets
The paper introduces a new dataset consisting of 1,608 Slack messages from the real-world interactions of a 7-member research group. This dataset was specifically curated to capture the "many-to-many" structure of group conversations, formatting each data point with a "context" (previous messages) and a "target" (the desired response). This differs from typical datasets that focus on one-to-one dialogues.
Key Insights and Ideas
- The Need for Multi-Actor AI: Traditional conversational AI falls short in group settings. The paper highlights the critical need for AI that can understand and participate in complex, multi-participant dialogues, not just simple Q&A.
- Fine-tuning for Context is Crucial: While pre-trained LLMs show promise, they struggle to capture the specific language style and structure of group conversations. Fine-tuning, even with efficient methods like QLoRA, is essential for aligning the AI with the nuances of team dynamics.
- Balancing AI Participation: An AI that is too active can disrupt group flow, while one that is too passive might be unhelpful. Finding the right balance in the AI's engagement is key to its effectiveness.
- Human-Centric Evaluation Matters: Quantitative metrics like BLEU and perplexity are useful, but qualitative feedback from users in realistic workshop settings is vital for understanding the real-world impact and identifying areas for improvement.
Key Results
- LLM Performance: In initial evaluations, pre-trained LLMs generally had lower perplexity but lower BLEU scores compared to fine-tuned models on the group conversation dataset. Mistral-7B-Instruct-v0.1 achieved the highest BLEU score (9.27%), indicating better syntactic similarity to the reference text.
- Fine-tuning Impact: The fine-tuned Mistral-7B-Instruct-v0.1 model showed a significant decrease in perplexity during training, reaching an average of 7.8 on the test set, suggesting it did not overfit. Subjectively, its responses were perceived as more natural, context-aware, and better at following instructions.
- Qualitative Feedback: Participants found the bot to be dynamic but sometimes too active and quick to respond, occasionally dominating conversations. Responses were often too long and detailed. The bot's informal, emoji-heavy tone was sometimes perceived as unnatural in a professional setting, and it occasionally made mistakes with names. It also struggled to provide critical feedback during brainstorming sessions.
Limitations and Future Work
The authors acknowledge several limitations:
- Dataset Size: The training dataset, while novel, is relatively small (1,608 messages) and comes from a single research group, potentially limiting generalization to other teams and contexts.
- Fine-tuning Technique: Using QLoRA, while efficient, might restrict the depth of learning compared to full fine-tuning.
- Evaluation Metrics: Perplexity and BLEU scores don't fully capture the nuances of conversation flow and engagement in multi-actor settings.
- Turn-Taking: Effervesce currently lacks sophisticated turn-taking regulation, which can disrupt natural conversation flow.
Future work will focus on:
- Increasing dataset size and diversity.
- Exploring newer open-source LLMs.
- Improving the bot's response strategy, potentially integrating tool-calling and Retrieval Augmented Generation (RAG).
- Developing more advanced turn-taking mechanisms to optimize engagement and response length/style.
- Ensuring the bot facilitates balanced participation and avoids overwhelming users.
Summary updated