LLMs Cannot (Yet) Match the Specificity and Simplicity of Online Communities in Long Form Question Answering

Kris-Fillip Kahl, Tolga Buz, Russa Biswas, Gerard de MeloEMNLP 2024 · DOI

Introduction

The landscape of financial education is undergoing a significant transformation. Traditionally, retail investors turned to massive online communities like Reddit’s r/wallstreetbets or r/personalfinance to navigate the complexities of the market. Today, many are instead querying Large Language Models (LLMs) like ChatGPT for financial guidance.

However, a notable "domain gap" exists. Standard instruction-tuned LLMs tend to produce "textbook" responses—long-winded, overly formal, and often missing the specific nuances of actual financial discourse. They frequently fail to match the specificity (addressing the exact query) and simplicity (using accessible language) found in successful human-to-human interactions. This research aims to bridge this gap by leveraging the collective intelligence of Reddit to align LLMs with how humans actually discuss finance.

The SOCIALFINANCEQA Dataset

To address this challenge, the researchers introduced SOCIALFINANCEQA, the first social-media preference dataset specifically curated for the finance domain. The dataset is derived from 15 of the largest finance-related subreddits, capturing the diverse linguistic styles of communities ranging from the high-risk environment of r/pennystocks to the long-term planning focus of r/financialindependence.

Statistics of the subreddits used to construct the SOCIALFINANCEQA dataset, showing millions of submissions and comments.

The scale of the raw data is immense, involving 7.4 million submissions and over 82 million comments spanning from 2008 to 2022. Through a multi-stage pipeline involving rigorous filters for engagement (upvotes), community approval, and toxicity scrubbing (using models like HateBERT), the researchers produced a high-quality set of 61,610 question-answer-answer tuples. In these tuples, one answer is community-preferred (the "better" answer) over another, providing a clear signal for preference optimization.

Methodology and Alignment

The researchers employed a two-stage training process to adapt base models like Llama-2, Mistral, and Zephyr to the financial domain:

  1. Supervised Fine-Tuning (SFT): This stage teaches the models the specific domain language and jargon used in financial communities.
  2. Direct Preference Optimization (DPO): This stage aligns the models with human preferences by training them to choose the style of highly upvoted Reddit comments over less popular ones.

To evaluate these models, the authors proposed SOCIALQA-EVAL, a framework that uses an "LLM-as-a-judge" (Gemini Pro) alongside human annotators to score responses across five critical dimensions: Relevance, Specificity, Simplicity, Helpfulness, and Objectivity.

Key Insights and Results

The "Length Bias" Conflict

A striking finding of the study is the disagreement between AI judges and human evaluators. LLM judges consistently preferred longer, more verbose "textbook" answers, often equating length with quality. In contrast, human evaluators preferred the fine-tuned models that were more concise and direct.

Performance Gains in Specificity and Simplicity

Fine-tuning on Reddit data significantly improved model performance in areas where generic LLMs typically struggle. Human annotators noted that fine-tuned models like Zephyr-DPO and Mistral-SFT-DPO were much more specific and simpler than their baseline counterparts. For instance, Llama-2’s overall score in human evaluation nearly doubled after being fine-tuned on the SOCIALFINANCEQA dataset.

Evaluation results comparing LLM-as-a-judge (LLM) and Human (HU) scores across five dimensions. Note the significant improvements in Specificity and Simplicity for fine-tuned models.

Readability and Perplexity

Fine-tuning also lowered the "reading level" (Flesch-Kincaid score) of the models. In finance, where jargon can be a significant barrier to entry, the ability to explain complex concepts simply is a major performance gain. Furthermore, SFT significantly reduced the models' perplexity on financial text, indicating that the models successfully learned the unique linguistic patterns of the online finance community.

Limitations and Future Directions

While the results are promising, several limitations remain. Due to Reddit API changes in mid-2023, the dataset only extends to the end of 2022, meaning it does not capture the most recent market shifts. Additionally, while the models become more "human-like," they are not certified financial advisors; there remains a risk of generating confident but incorrect guidance.

Future work may involve expanding the SOCIALQA-EVAL framework to other specialized domains, such as medicine or law, where community-specific simplicity and specificity are equally vital for effective communication.

Summary updated

BibTeX
@inproceedings{kahl2024llms,
  author = {Kris-Fillip Kahl and Tolga Buz and Russa Biswas and Gerard de Melo},
  title = {{LLMs Cannot (Yet) Match the Specificity and Simplicity of Online Communities in Long Form Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.111},
}
Figure