WildVis: Open Source Visualizer for Million-Scale Chat Logs in the Wild

Yuntian Deng, Wenting Zhao, Jack Hessel, Xiang Ren, Claire Cardie, Yejin Choi

Abstract

WildVis is an open-source interactive tool designed to help researchers navigate and analyze million-scale chatbot interaction datasets. By combining a high-performance metadata search engine with a semantic embedding visualizer, it allows users to uncover patterns, identify misuse, and compare topic distributions across massive datasets like WildChat and LMSYS-Chat-1M in sub-second time.

Introduction

The release of massive datasets like WildChat and LMSYS-Chat-1M has provided a goldmine for studying how humans interact with AI. However, these datasets contain millions of conversations, making manual inspection impossible. Researchers face a "needle in a haystack" problem: finding specific instances of chatbot misuse, identifying regional variations in prompts, or understanding the semantic landscape of a million conversations without spending weeks on data processing.

Existing tools are often too generic or lack the scalability to handle the unique metadata and volume of conversational logs. WildVis bridges this gap by providing a specialized interface that handles both exact filtering and exploratory semantic visualization at scale.

System Architecture and Methods

WildVis operates on a client-server architecture designed for low latency. The system consists of two primary modules that work in tandem to provide a comprehensive view of the data.

Built on Elasticsearch, this module allows users to compose complex queries using ten predefined filters. These include keywords, geographical location (state/country), hashed IP addresses (to track user-specific patterns), toxicity flags, and PII (Personally Identifiable Information) redaction status.

2. Embedding Visualization

To represent the "meaning" of conversations, the authors embed the first user turn of every conversation using OpenAI’s text-embedding-3-small model. These high-dimensional vectors are then projected into a 2D plane using Parametric UMAP.

System architecture of WildVis, detailing the interaction between the frontend, Flask backend, and Elasticsearch engine.

To maintain responsiveness, the team implemented several optimizations:

  • Precomputation & Compression: 2D coordinates are precomputed and stored in a compressed 1MB file that users download once.
  • Caching: Results for dynamic queries are cached in an SQLite database.
  • Frontend Rendering: The tool uses Deck.gl to efficiently render thousands of points in the browser, ensuring smooth zooming and panning even on mobile devices.

Datasets and Semantic Insights

The tool currently supports two major open-source datasets: WildChat (1 million real-world interactions) and LMSYS-Chat-1M (1 million conversations from the Chatbot Arena). WildVis is designed to be extendable, allowing researchers to plug in additional datasets by following the authors' indexing and embedding pipeline.

Semantic "Neighborhoods"

The embedding map naturally clusters conversations by topic. For example, WildVis reveals a distinct cluster in WildChat dedicated to "Midjourney prompt engineering," a phenomenon less prevalent in the LMSYS dataset.

The WildVis embedding visualization interface, illustrating semantic clusters and specific filter matches across datasets.

User Tracking and Language Specificity

By making hashed IP addresses clickable, the tool allows researchers to instantly pivot from a single suspicious conversation to the entire history of that specific (anonymized) user. This led to the discovery of a "Visa Officer" case where a user appeared to be disclosing sensitive client information to the bot.

Furthermore, the authors found that training a single UMAP model on all languages led to clusters driven by language rather than topic. To fix this, they trained separate parametric UMAP models for different languages, allowing for more semantically meaningful exploration within specific linguistic contexts (e.g., English, Chinese, Russian).

Performance Results

The primary metric for WildVis is efficiency. In performance evaluations, the system demonstrated significant improvements over standard data processing methods:

  • Search Speed: A keyword-based search in WildVis takes an average of 0.47 seconds. For comparison, a naive approach using the HuggingFace Datasets library took over 19 minutes (1148 seconds) for the same query.
  • Visualization Speed: Rendering and highlighting matching conversations on the embedding map takes roughly 0.43 seconds.
  • Scalability: The tool successfully manages 2 million conversations while keeping the frontend payload small (approx. 1MB for precomputed coordinates).

Limitations and Future Work

While powerful, WildVis has limitations. Currently, only the first user turn is used for embeddings, which may miss shifts in topic that occur later in long, multi-turn conversations. Additionally, WildVis is primarily an exploratory tool; while it helps form hypotheses, researchers still need to perform separate quantitative analyses for final results. Future work includes expanding to more specialized datasets and incorporating multi-turn embedding analysis.

WildVis is open-source and available at wildvisualizer.com.

Summary updated

Abstract

The increasing availability of real-world conversation data offers exciting opportunities for researchers to study user-chatbot interactions. However, the sheer volume of this data makes manually examining individual conversations impractical. To overcome this challenge, we introduce WildVis, an interactive tool that enables fast, versatile, and large-scale conversation analysis. WildVis provides search and visualization capabilities in the text and embedding spaces based on a list of criteria. To manage million-scale datasets, we implemented optimizations including search index construction, embedding precomputation and compression, and caching to ensure responsive user interactions within seconds. We demonstrate WildVis’ utility through three case studies: facilitating chatbot misuse research, visualizing and comparing topic distributions across datasets, and characterizing user-specific conversation patterns. WildVis is open-source and designed to be extendable, supporting additional datasets and customized search and visualization functionalities.
BibTeX
@inproceedings{deng2024wildvis,
  author = {Yuntian Deng and Wenting Zhao and Jack Hessel and Xiang Ren and Claire Cardie and Yejin Choi},
  title = {{WildVis: Open Source Visualizer for Million-Scale Chat Logs in the Wild}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=ir6WvAVIZ3},
}
Figure