HelloFresh: LLM Evaluations on Streams of Real-World Human Editorial Actions across X Community Notes and Wikipedia edits

Tim Franzmeyer, Aleksandar Shtedritski, Samuel Albanie, Philip Torr, João F. Henriques, Jakob N. Foerster

TL;DR

HelloFresh is a new "living" benchmark that evaluates Large Language Models (LLMs) using continuous streams of real-world editorial data from X (formerly Twitter) Community Notes and Wikipedia edits. By leveraging data generated by intrinsically motivated human contributors after the models' training cutoffs, it provides a temporally consistent ranking of model performance while effectively mitigating the risks of test data contamination and benchmark overfitting.

What Problem Does This Paper Address?

Static benchmarks have long been the yardstick for AI progress, but they face three growing crises:

  1. Data Contamination: Because LLMs are trained on massive web scrapes, they often "see" the test questions during training, leading to inflated performance scores.
  2. Benchmark Overfitting: Researchers may inadvertently optimize models for specific, unchanging datasets rather than general intelligence.
  3. Lack of Real-World Grounding: Many benchmarks rely on artificial tasks or paid annotators who lack the "intrinsic motivation" of real users, potentially resulting in lower-quality labels.

The authors introduce HelloFresh to solve these issues by using dynamic, community-adjudicated data that reflects how humans actually verify information and reach consensus in the wild.

Methods

The researchers developed two primary evaluation regimes:

  • Zero-shot Classification: Models are given a social media post and a community note (or a Wikipedia paragraph and a proposed edit) and must decide if the note is "helpful" or the edit is "correct."
  • Web-search Agent: To simulate real-world fact-checking, this regime allows the LLM to generate a search query, retrieve and summarize the top five web results, and then use that external context to make its classification.

To ensure the results weren't just a fluke of specific wording, they used four different LLM-generated rephrasings for every prompt. For Wikipedia, they developed a specific algorithm to determine "ground truth": an edit is considered "accepted" only if the affected paragraph remains unchanged for a calculated number of subsequent edits, signaling community consensus.

The algorithm used to filter and classify Wikipedia edits as 'accepted' or 'rejected'.

Datasets

The benchmark introduces two new, periodically updated datasets:

  • X Community Notes: ~3,000 samples (Oct 2023 – Jan 2024) consisting of posts and notes rated by the community.
  • Wikipedia Edits: ~4,000 samples (Feb – March 2024) tracking whether edits were accepted or eventually reverted.

These datasets are unique because they are "living"—the authors host a public leaderboard and release new, unlabeled data quarterly to ensure models are always tested on events that occurred after their training data was finalized.

Key Insights and Ideas

  • Temporal Consistency is Key: One of the most important findings is that despite the data evolving over time, the relative ranking of models remains stable. This suggests that HelloFresh is a reliable metric for tracking progress.
F1 scores over time show that model rankings remain largely consistent across different months and weeks.
  • The "Confusion" Paradox: Counterintuitively, adding web search results sometimes decreased performance for models like GPT-3.5 and Mistral. The authors found that these models could become "confused" by the large volume of information, even if it was relevant.
  • Prompt Sensitivity: Performance varied significantly based on how the task was phrased, highlighting the critical need for prompt tuning in real-world deployments.
  • Consensus Strength: Models performed significantly better on X notes that had received a high volume of community votes, suggesting that LLMs align better with strong human consensus than with disputed or low-engagement content.

Key Results

  • Top Performers: GPT-4 consistently led the pack on X Community Notes, while GPT-3.5 showed surprising strength in the zero-shot Wikipedia task.
  • Deployment Readiness: In a simulated deployment scenario using prompt tuning, the best models achieved 90% precision on Wikipedia edits and 75% precision on X notes, provided they maintained a recall of at least 20%.
  • Search Impact: While web search helped GPT-4 on X notes, it led to a 7.3% to 8.9% drop in F1 score for GPT-3.5 and Mistral-8x7 on Wikipedia tasks compared to the zero-shot approach.
The difference in F1 score between zero-shot and web-search regimes across different prompts.

Limitations and Future Work

The authors acknowledge that "ground truth" on these platforms is based on human consensus, which can be subjective or reflect the biases of the user base (e.g., X's predominantly male demographic).

Future work aims to:

  • Expand to Multi-modal: Incorporating images and videos, which are central to many misleading social media posts.
  • Generative Tasks: Moving beyond verification to asking LLMs to propose helpful notes or Wikipedia edits.
  • Adversarial Robustness: Evaluating how these community mechanisms (and the LLMs evaluating them) hold up against coordinated misinformation campaigns.

Summary updated

Abstract

Benchmarks have been essential for driving progress in machine learning. A better understanding of LLM capabilities on real world tasks is vital for safe development. Designing adequate LLM benchmarks is challenging: Data from real-world tasks is hard to collect, public availability of static evaluation data results in test data contamination and benchmark overfitting, and periodically generating new evaluation data is tedious and may result in temporally inconsistent results. We introduce HelloFresh, based on continuous streams of real-world data generated by intrinsically motivated human labelers. It covers recent events from X (formerly Twitter) community notes and edits of Wikipedia pages, mitigating the risk of test data contamination and benchmark overfitting. Any X user can propose an X note to add additional context to a misleading post (formerly tweet); if the community classifies it as helpful, it is shown with the post. Similarly, Wikipedia relies on community-based consensus, allowing users to edit articles or revert edits made by other users. Verifying whether an X note is helpful or whether a Wikipedia edit should be accepted are hard tasks that require grounding by querying the web. We backtest state-of-the-art LLMs supplemented with simple web search access and find that HelloFresh yields a temporally consistent ranking. To enable continuous evaluation on HelloFresh, we host a public leaderboard and periodically updated evaluation data at https://tinyurl.com/hello-fresh-LLM.
BibTeX
@inproceedings{franzmeyer2024hellofresh,
  author = {Tim Franzmeyer and Aleksandar Shtedritski and Samuel Albanie and Philip Torr and João F. Henriques and Jakob N. Foerster},
  title = {{HelloFresh: LLM Evaluations on Streams of Real-World Human Editorial Actions across X Community Notes and Wikipedia edits}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.754},
}
Figure