Investigating Thematic Patterns and User Preferences in LLM Interactions using BERTopic

Abhay Bhandarkar, Gaurav Mishra, Khushi Juchani, Harsh SinghalarXiv 2025 · arXiv · OpenReview

TL;DR

This paper uses BERTopic, a powerful topic modeling technique, to analyze a massive dataset of human-LLM conversations. The goal is to uncover the underlying themes users discuss with LLMs and see how user preferences for different LLMs vary across these themes, offering insights into LLM strengths and weaknesses.

What Problem Does This Paper Address?

The rapid proliferation of Large Language Models (LLMs) has created a critical need to understand how humans interact with them. While LLMs are becoming increasingly sophisticated, it's challenging to pinpoint their specific strengths and weaknesses across the vast landscape of user queries. Existing evaluation methods often lack a deep thematic analysis, making it hard to understand why users prefer one LLM over another for particular tasks. This paper aims to bridge this gap by identifying thematic patterns in LLM conversations and correlating them with human preferences, providing a more nuanced view of LLM performance.

Methods

The core of this research lies in applying BERTopic, a state-of-the-art topic modeling technique. BERTopic leverages transformer-based embeddings (specifically, all-MiniLM-L6-v2) to represent conversations as dense vectors. These vectors are then dimensionality-reduced using UMAP to preserve semantic structure while making them computationally tractable. Finally, HDBSCAN is used for density-based clustering to identify coherent topics without needing to pre-specify the number of topics. Topic representation is achieved using a class-based TF-IDF (c-TF-IDF) approach to extract descriptive keywords for each topic. The authors also detail a meticulous preprocessing pipeline to clean and normalize the conversational data.

Datasets

The primary dataset used is the LMSYS-Chat-1M dataset. This dataset was collected between April and August 2023 from the Vicuna demo and Chatbot Arena, comprising approximately 210,000 unique IP addresses. It contains multilingual conversational data, including user prompts, two anonymized LLM responses, and human preference labels indicating which response was preferred. This dataset is valuable as it represents real-world LLM interactions and human evaluations.

Key Insights and Ideas

  • LLMs excel in specific domains: No single LLM is a universal champion. Different LLMs demonstrate significantly stronger performance in particular thematic areas, suggesting that users implicitly choose models suited to their needs.
  • User preference for conciseness: Shorter responses were preferred by users more often than longer ones (57.9% vs. 42.1%), indicating a general user tendency towards brevity.
  • Dominant topics drive engagement: A small subset of the identified topics (the top 10) accounts for a substantial majority (over 78.3%) of user interactions, highlighting areas of high user interest.
  • Niche strengths are important: The analysis revealed specific LLMs showing exceptional performance in niche topics, which might be overlooked in broader evaluations. For example, one model excelled in "Social Issues and Ethical Dilemmas," while another showed strength in "Science & Astronomy."

Key Results

  • 29 distinct thematic clusters were identified using BERTopic, covering a wide range of subjects from gaming and programming to politics and health.
  • The top 10 topics cumulatively cover over 78.3% of the analyzed user interactions.
  • gpt-4-1106-preview and gpt-3.5-turbo-0613 were among the top-performing models across many topics.
  • gpt-3.5-turbo-0314 showed the highest "balanced" win rate, meaning it performed well proportionally to its number of appearances.
  • Specific models like gpt-4-0314 demonstrated very high win rates in specific topics (e.g., 49.97% in "Social Issues and Ethical Dilemmas").

Limitations and Future Work

  • Outlier Handling: While HDBSCAN identified outliers (Topic -1), these were excluded from the main topic interpretation to maintain coherence. The authors note that these outliers represent a high degree of lexical and semantic diversity.
  • Multilingual Data: Although the dataset is multilingual, the analysis was confined to English-language interactions due to preprocessing limitations.
  • Future Work: The authors suggest generalizing the topic-centric approach to multimodal inputs (including vision-based tasks) and further investigating topical balance within conversational AI systems. They also propose empowering developers to build more versatile and adaptive AI systems by understanding domain-specific LLM strengths.

Summary updated

Abstract

This study applies BERTopic, a transformer-based topic modeling technique, to the lmsys-chat-1m dataset, a multilingual conversational corpus built from head-to-head evaluations of large language models (LLMs). Each user prompt is paired with two anonymized LLM responses and a human preference label, used to assess user evaluation of competing model outputs. The main objective is uncovering thematic patterns in these conversations and examining their relation to user preferences, particularly if certain LLMs are consistently preferred within specific topics. A robust preprocessing pipeline was designed for multilingual variation, balancing dialogue turns, and cleaning noisy or redacted data. BERTopic extracted over 29 coherent topics including artificial intelligence, programming, ethics, and cloud infrastructure. We analysed relationships between topics and model preferences to identify trends in model-topic alignment. Visualization techniques included inter-topic distance maps, topic probability distributions, and model-versus-topic matrices. Our findings inform domain-specific fine-tuning and optimization strategies for improving real-world LLM performance and user satisfaction.
BibTeX
@misc{bhandarkar2025investigating,
  author = {Abhay Bhandarkar and Gaurav Mishra and Khushi Juchani and Harsh Singhal},
  title = {{Investigating Thematic Patterns and User Preferences in LLM Interactions using BERTopic}},
  year = {2025},
  eprint = {2510.07557},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.07557},
}
Figure