CoSearchAgent: A Lightweight Collaborative Search Agent with Large Language Models

Peiyuan Gong, Jiamian Li, Jiaxin Mao

Introduction

Collaborative search is a common practice where multiple individuals coordinate to solve complex information tasks. Despite the prevalence of dedicated tools, most users prefer to conduct these activities within instant messaging (IM) platforms like Slack or Microsoft Teams. However, search queries in group chats are often "lightweight," fragmented, and heavily dependent on conversational context. For example, a user might request "more info on that," referring to a topic discussed much earlier in the thread.

Historically, bridging this gap required a "Wizard of Oz" approach, where human operators manually interpreted context and provided results. CoSearchAgent addresses this by leveraging the reasoning capabilities of Large Language Models (LLMs) to automate the entire collaborative search pipeline within existing IM environments.

System Architecture

CoSearchAgent is implemented as a Slack plugin using the Bolt-Python framework and GPT-3.5-turbo. The system operates through a three-stage pipeline designed to handle the nuances of multi-party dialogue.

The three-stage pipeline of CoSearchAgent: Query Processing, Result Presentation, and Retrieval-Augmented Generation.
  1. Query Processing: Upon being mentioned, the agent analyzes the preceding 20 messages to rewrite the user's request into a standalone, search-engine-optimized query. If ambiguity persists, the agent proactively asks a clarifying question.
  2. Search Result Presentation: Utilizing the Google Search API, the agent retrieves top results. Rather than displaying generic snippets, it crawls the HTML content (up to 5000 tokens) and uses the LLM to extract and summarize only the information relevant to the specific group query.
  3. Retrieval-Augmented Generation (RAG): The final stage synthesizes a comprehensive answer based on the extracted snippets. This response includes inline citations (e.g., [1], [2]) linking to source materials, which mitigates hallucinations and builds user trust.

Data Collection and Logging

While CoSearchAgent is a functional system rather than a pre-trained model, it serves as a powerful tool for generating research datasets. The system implements a comprehensive logging framework that captures multi-party dialogue history, generated search queries, retrieved results, and user click logs for future analysis.

Discussion and Key Insights

The development of CoSearchAgent highlighted several critical factors for successful collaborative AI assistants. First, contextual rewriting is essential; ignoring the chat history leads to search failure in group settings. By expanding queries based on history, the agent achieves semantic consistency comparable to human researchers.

Second, snippet replacement significantly improves the user experience. By replacing SEO-optimized snippets with LLM-generated summaries, users can evaluate relevance without leaving the chat interface. Finally, the agent's mixed-initiative interaction—asking for clarification rather than guessing—reduces the trial-and-error loop typical of complex searches.

Evaluation and Results

The authors evaluated CoSearchAgent by comparing it against three alternative modes: Direct Search, a human-led "Wizard of Oz" setup, and a version with Query Processing but no RAG.

Comparison of search modes: Direct Search, Wizard of Oz, Query Processing + Search, and the full CoSearchAgent system.

The case study revealed that CoSearchAgent matches the accuracy of human "wizards" in query rewriting. Qualitatively, the RAG-based answers with citations were found to be superior to raw search results, providing direct answers while maintaining verifiability. The system demonstrates that LLMs can effectively manage the "heavy lifting" of multi-user interactions that previously necessitated human intervention.

Limitations and Future Work

Current limitations include the LLM's context window, which may restrict performance in exceptionally long research sessions. Additionally, the agent is currently limited to a single round of clarification to avoid intrusiveness. Future research will focus on large-scale user studies to quantify time savings and optimize the balance between agent proactivity and user experience.

Summary updated

Abstract

Collaborative search supports multiple users working together to accomplish a specific search task. Research has found that designing lightweight collaborative search plugins within instant messaging platforms aligns better with users' collaborative habits. However, due to the complexity of multi-user interaction scenarios, it is challenging to implement a fully functioning lightweight collaborative search system. Therefore, previous studies on lightweight collaborative search had to rely on the Wizard of Oz paradigm. In recent years, large language models (LLMs) have been demonstrated to interact naturally with users and achieve complex information-seeking tasks through LLM-based agents. Hence, to better support the research in collaborative search, in this demo, we propose CoSearchAgent, a lightweight collaborative search agent powered by LLMs. CoSearchAgent is designed as a Slack plugin that can support collaborative search during multi-party conversations on this platform. Equipped with the capacity to understand the queries and context in multi-user conversations and the ability to search the Web for relevant information via APIs, CoSearchAgent can respond to user queries with answers grounded on the relevant search results. It can also ask clarifying questions when the information needs are unclear. The proposed CoSearchAgent is highly flexible and would be useful for supporting further research on collaborative search. The code and demo video are accessible.
BibTeX
@inproceedings{gong2024cosearchagent,
  author = {Peiyuan Gong and Jiamian Li and Jiaxin Mao},
  title = {{CoSearchAgent: A Lightweight Collaborative Search Agent with Large Language Models}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2024},
  doi = {10.1145/3626772.3657672},
}
Figure