TL;DR
Training personalized information retrieval (PIR) systems is notoriously difficult due to data scarcity and user privacy concerns. This paper demonstrates that synthetic data generated by Large Language Models (LLMs) can successfully replace human-written data to train effective neural re-rankers for community question answering, surprisingly revealing that a high rate of LLM hallucinations does not hinder downstream retrieval performance.
What Problem Does This Paper Address?
Personalized Information Retrieval (PIR) aims to tailor search results to an individual's specific interests and preferences. In the context of community Question Answering (cQA) platforms like StackExchange, personalization helps surface the most relevant answers based on a user's history and community context.
However, developing deep learning models for PIR faces a major bottleneck: the lack of large-scale, publicly available training datasets. Collecting high-quality user data is expensive, time-consuming, and heavily restricted by privacy regulations and anonymization needs. To bypass this data drought, the authors investigate whether LLMs can generate high-quality synthetic documents tailored to user needs, effectively serving as a drop-in replacement for human-authored training data.
Methods
The authors design a two-stage retrieval pipeline to evaluate the utility of synthetic data:
- First-Stage Retrieval: A standard BM25 retriever pulls the top 100 candidate answers from a pool.
- Second-Stage Re-ranking: A neural re-ranker (DistilBERT) scores and re-ranks these candidates. The final score is a convex combination ($\lambda$) of the BM25 and DistilBERT scores.
The core novelty lies in how the DistilBERT model is trained. Instead of using real human answers, it is fine-tuned on synthetic answers generated by three different LLMs: GPT-3.5 Turbo, Phi-3-mini (3.8B parameters), and Phi-3-medium (14B parameters).
To test the impact of personalization and context, they employ three distinct prompting strategies:
- Basic: The LLM answers the question using only its title and body.
- Personalized: The prompt includes the user's top five most frequently used tags to simulate individual user interests.
- Contextual: The prompt embeds the specific StackExchange community context (e.g., "History" vs. "Movies") to shape the domain flavor of the answer.
Datasets
The study utilizes and introduces two main datasets:
- SE-PQA: An existing dataset containing over 200k training questions across 50 different StackExchange communities, complete with user tags and community metadata.
- Sy-SE-PQA (New): The newly introduced synthetic dataset containing approximately 100k answers per prompt type for each of the three LLMs, built by sampling up to 3,000 questions per community from SE-PQA.
Key Insights and Ideas
- The Hallucination Paradox: A manual evaluation revealed that over 35% of the LLM-generated answers contained factual errors or hallucinations. Crucially, this did not degrade the retrieval performance of the fine-tuned DistilBERT model. For information retrieval, the semantic alignment and vocabulary matching provided by synthetic data matter more than strict factual accuracy.
- Small Models, Big Impact: Open-source Small Language Models (SLMs) like Phi-3-mini and Phi-3-medium actually generated synthetic data that led to better downstream IR performance than the larger, closed-source GPT-3.5 Turbo.
- Lexical Overlap vs. Quality: LLMs tend to repeat query words in their responses, leading to higher lexical overlap (around 33-35%) compared to human answers (23.4%). However, higher lexical overlap isn't the sole driver of performance; the contextual and personalized nuances introduced via prompting significantly boosted ranking metrics.
Key Results
The neural re-ranker trained on synthetic data consistently outperformed the baseline BM25 model and matched or exceeded the performance of the model trained on real human answers when evaluated on a human-annotated test set.
- Phi-3-mini Performance: When DistilBERT was trained on personalized synthetic data from Phi-3-mini and combined with BM25, P@1 jumped from 0.279 (BM25 baseline) to 0.345, outperforming the "Real Answer" baseline (0.333).
- Phi-3-medium Performance: Achieved the highest overall metrics when utilizing personalized prompts, reaching a P@1 of 0.347 and an NDCG@10 of 0.468.
- GPT-3.5 Performance: Showed a slight performance drop when moving from basic to personalized prompts, indicating that smaller models might be more robust to this specific style of synthetic data generation.
Limitations and Future Work
- Bias and Fairness: Synthetic data inherits the inherent societal prejudices and biases present in the underlying LLMs' training sets, which could lead to discriminatory retrieval behavior.
- Scale Constraints: Due to computational and budget limitations, data generation was capped at 3,000 questions per community, potentially limiting data diversity.
- Evaluation Metrics: The study relies purely on traditional IR metrics (P@1, NDCG, MAP). These metrics do not measure qualitative aspects like readability, coherence, or user satisfaction.
- Future Directions: The authors plan to explore more sophisticated prompting techniques that utilize a broader array of user features and investigate Retrieval-Augmented Generation (RAG) to mitigate the high hallucination rates.
Summary updated