TL;DR
HelloFresh is a new "living" benchmark that evaluates Large Language Models (LLMs) using continuous streams of real-world editorial data from X (formerly Twitter) Community Notes and Wikipedia edits. By leveraging data generated by intrinsically motivated human contributors after the models' training cutoffs, it provides a temporally consistent ranking of model performance while effectively mitigating the risks of test data contamination and benchmark overfitting.
What Problem Does This Paper Address?
Static benchmarks have long been the yardstick for AI progress, but they face three growing crises:
- Data Contamination: Because LLMs are trained on massive web scrapes, they often "see" the test questions during training, leading to inflated performance scores.
- Benchmark Overfitting: Researchers may inadvertently optimize models for specific, unchanging datasets rather than general intelligence.
- Lack of Real-World Grounding: Many benchmarks rely on artificial tasks or paid annotators who lack the "intrinsic motivation" of real users, potentially resulting in lower-quality labels.
The authors introduce HelloFresh to solve these issues by using dynamic, community-adjudicated data that reflects how humans actually verify information and reach consensus in the wild.
Methods
The researchers developed two primary evaluation regimes:
- Zero-shot Classification: Models are given a social media post and a community note (or a Wikipedia paragraph and a proposed edit) and must decide if the note is "helpful" or the edit is "correct."
- Web-search Agent: To simulate real-world fact-checking, this regime allows the LLM to generate a search query, retrieve and summarize the top five web results, and then use that external context to make its classification.
To ensure the results weren't just a fluke of specific wording, they used four different LLM-generated rephrasings for every prompt. For Wikipedia, they developed a specific algorithm to determine "ground truth": an edit is considered "accepted" only if the affected paragraph remains unchanged for a calculated number of subsequent edits, signaling community consensus.
Datasets
The benchmark introduces two new, periodically updated datasets:
- X Community Notes: ~3,000 samples (Oct 2023 – Jan 2024) consisting of posts and notes rated by the community.
- Wikipedia Edits: ~4,000 samples (Feb – March 2024) tracking whether edits were accepted or eventually reverted.
These datasets are unique because they are "living"—the authors host a public leaderboard and release new, unlabeled data quarterly to ensure models are always tested on events that occurred after their training data was finalized.
Key Insights and Ideas
- Temporal Consistency is Key: One of the most important findings is that despite the data evolving over time, the relative ranking of models remains stable. This suggests that HelloFresh is a reliable metric for tracking progress.
- The "Confusion" Paradox: Counterintuitively, adding web search results sometimes decreased performance for models like GPT-3.5 and Mistral. The authors found that these models could become "confused" by the large volume of information, even if it was relevant.
- Prompt Sensitivity: Performance varied significantly based on how the task was phrased, highlighting the critical need for prompt tuning in real-world deployments.
- Consensus Strength: Models performed significantly better on X notes that had received a high volume of community votes, suggesting that LLMs align better with strong human consensus than with disputed or low-engagement content.
Key Results
- Top Performers: GPT-4 consistently led the pack on X Community Notes, while GPT-3.5 showed surprising strength in the zero-shot Wikipedia task.
- Deployment Readiness: In a simulated deployment scenario using prompt tuning, the best models achieved 90% precision on Wikipedia edits and 75% precision on X notes, provided they maintained a recall of at least 20%.
- Search Impact: While web search helped GPT-4 on X notes, it led to a 7.3% to 8.9% drop in F1 score for GPT-3.5 and Mistral-8x7 on Wikipedia tasks compared to the zero-shot approach.
Limitations and Future Work
The authors acknowledge that "ground truth" on these platforms is based on human consensus, which can be subjective or reflect the biases of the user base (e.g., X's predominantly male demographic).
Future work aims to:
- Expand to Multi-modal: Incorporating images and videos, which are central to many misleading social media posts.
- Generative Tasks: Moving beyond verification to asking LLMs to propose helpful notes or Wikipedia edits.
- Adversarial Robustness: Evaluating how these community mechanisms (and the LLMs evaluating them) hold up against coordinated misinformation campaigns.
Summary updated