QuaLLM: An LLM-based Framework to Extract Quantitative Insights from Online Forums

Varun Nagaraj Rao, Eesha Agarwal, Samantha Dalal, Dana Calacci, Andrés Monroy-HernándezNAACL 2025 · DOI

TL;DR

This paper introduces QuaLLM, a novel framework that leverages Large Language Models (LLMs) to extract quantitative insights from vast amounts of unstructured text data found in online forums. By automating thematic analysis and providing quantifiable metrics, QuaLLM addresses the scalability limitations of traditional methods, enabling deeper understanding of community concerns.

What Problem Does This Paper Address?

Online discussion forums, like Reddit, are rich sources of information for understanding diverse real-world communities. However, analyzing this data at scale is challenging. Traditional qualitative methods like thematic analysis are labor-intensive and difficult to scale. Quantitative methods like topic modeling (e.g., LDA) offer scalability but require careful hyperparameter tuning, have limited context windows, and still demand significant human effort to interpret outputs into human-readable themes. This gap hinders researchers from efficiently extracting actionable insights from the ever-growing volume of online discussions.

Methods

The core of this paper is the QuaLLM framework, which employs a novel prompting and human evaluation methodology. It's a multi-phase process designed to mimic human analytical steps:

  • Generation Prompt: This phase uses LLMs to generate summaries of concerns from online discussions, linking each concern to a representative quote. It involves several steps to ensure accuracy, avoid redundancy, and format outputs in JSON.
  • Classification Prompt: Identified concerns are then classified into predefined top-level themes (e.g., transparency, fairness). This acts as a filtration mechanism.
  • Aggregation Prompt: Concerns within each top-level theme are grouped into sub-themes, and representative concerns (top 5) are identified.
  • Prevalence Prompt: This final step quantifies the prevalence of concerns within each theme and sub-theme, providing a structured, survey-style output.

The framework also incorporates a robust evaluation strategy that mixes human and computational methods to assess factuality, completeness, accuracy, distinctness, and coverage of the LLM outputs.

Datasets

The primary dataset used for the case study is from Reddit's rideshare communities (r/uberdrivers and r/lyftdrivers). The study analyzed over 1 million comments from 2019-2022, comprising 65,377 submissions and 1,392,776 associated comments. This is noted as the largest study of its kind on this topic.

Key Insights and Ideas

  • LLMs as a Scalable Alternative: QuaLLM demonstrates that LLMs can effectively automate complex analytical tasks like thematic analysis, overcoming the scalability limitations of traditional human-driven methods.
  • Bridging Qualitative and Quantitative: The framework successfully translates unstructured qualitative data into a structured, quantitative format, making it easier to identify trends and concerns across large populations.
  • Surfacing Worker Concerns: The application to rideshare drivers highlights significant concerns regarding AI and algorithmic decisions, directly addressing calls for worker insights.
  • Mimicking Human Analysis: The multi-phase prompting strategy is designed to emulate the steps a human analyst would take, increasing the likelihood of generating meaningful and accurate insights.

Key Results

The study applied QuaLLM to over 1 million rideshare worker comments and identified key concerns categorized into four main themes:

  • Enhancing Transparency and Explainability: Drivers are concerned about opaque fare calculations, unclear incentives, and uncertain criteria for earnings (42% of concerns).
  • Predictability and Worker Agency: Drivers face unpredictable earnings due to fluctuating surge pricing, algorithm changes, and competition (22% of concerns).
  • Better Safety and More Time: Concerns include navigation issues, support difficulties, and low compensation for additional tasks (10.5% of concerns).
  • Ensuring Fairness and Non-Discrimination: Drivers dispute fare shares, platform take rates, and fairness in ride assignments (7% of concerns).

Quantitative results from the evaluation phase include:

  • Factuality: 0.55
  • Completeness: 0.78
  • Classification Accuracy: 0.74
  • Prevalence Accuracy: 0.82

The financial expenditure for processing this large dataset using GPT4-Turbo was $1,662.30.

Limitations and Future Work

  • Interpretivist vs. Positivist Trade-offs: While QuaLLM excels at aggregating broad trends (positivist approach), it may neglect individual experiences that are central to interpretivist qualitative research.
  • Human Evaluation Scale: The human evaluation was conducted on a relatively small sample size, limiting the generalizability of those specific findings.
  • Hyperparameter Sensitivity: BERTopic, used for aggregation, is sensitive to hyperparameter choices, which can lead to variations in results.
  • Reproducibility: The stochastic nature of LLM outputs raises concerns about reproducibility, though the authors note this mirrors human subjectivity.
  • Bias in LLMs: LLM outputs can be influenced by biases from training data and prompt design.
  • Misinformation: While misinformation in online communities can distort analysis, the authors suggest that aggregating over millions of posts may smooth out its impact.

The authors suggest that further research is essential to explore whether increased data volume truly equates to enhanced understanding and to investigate the broader implications of LLMs in quantitative text analysis. They also propose that QuaLLM can be integrated into mixed-methods study designs.

Summary updated

BibTeX
@inproceedings{rao2025quallm,
  author = {Varun Nagaraj Rao and Eesha Agarwal and Samantha Dalal and Dana Calacci and Andrés Monroy-Hernández},
  title = {{QuaLLM: An LLM-based Framework to Extract Quantitative Insights from Online Forums}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.74},
}
Figure