Introduction
Wikipedia is maintained by a massive community of volunteers who perform millions of edits every month. Ideally, every edit should include an edit summary—a short note explaining what was changed and why. These summaries are the frontline of defense for content moderators (patrollers) who use them to quickly distinguish between helpful updates and vandalism.
However, there is a major data gap: nearly 46% of Wikipedia edits have no summary at all. Even when provided, many summaries are "canned" (e.g., "fixed typo") or misleading. While large commercial models like GPT-4 are excellent at summarizing these changes, they are too expensive and resource-heavy to run at Wikipedia's scale. The challenge was to create a model that is small, fast, and accurate enough to assist editors in real-time.
Methodology
The researchers developed Edisum, a compact and efficient language model designed to automatically generate summaries for Wikipedia edits. They took a "distillation" approach to build a model that balances performance with efficiency:
- Qualitative Analysis: They first analyzed 100 human-written summaries to identify flaws, finding that humans often focus on the "what" but struggle to explain the "why" without external context.
- Synthetic Data Generation: To overcome the "mixed-quality" of human data, they used GPT-3.5-turbo to generate high-quality, consistent edit summaries for 100,000 edits. They used a 5-shot prompting strategy, providing the model with "edit diffs" (the specific lines added or removed).
- Model Architecture: They chose LongT5-local-base (~220M parameters). This model is small enough to run on CPUs (crucial for Wikipedia’s infrastructure) but features a long context window, which is necessary because edit diffs can be quite lengthy.
- Fine-tuning: They fine-tuned several versions of the model (Edisum) using different ratios of human-written vs. synthetic data to see which performed best.
The primary dataset consisted of English Wikipedia edits from August 2023, filtered for "non-trivial" changes to arrive at a clean set of ~127k samples.
Results and Key Insights
One of the most striking findings was that the model trained entirely on synthetic data (Edisum[100%]) performed better than the one trained on real human summaries. Human summaries are often noisy, inconsistent, or use obscure abbreviations; the LLM-generated data provided a "cleaner" signal for the small model to learn from.
In human head-to-head evaluations, Edisum[100%] performed on par with human editors. It was chosen as the "best" summary roughly as often as human-written ones. While massive models like GPT-4 remain the "gold standard" and outclass both humans and Edisum, the results demonstrate that a tiny 220M parameter model, when fine-tuned correctly on a specific task, can match the quality of average human editors.
Furthermore, Edisum is designed to run in real-time on CPUs, making it a viable tool for integration into the Wikipedia editing interface.
Limitations and Future Work
Despite its success, Edisum has limitations. The model is restricted by the information in the "diff." If the reason for an edit is based on a real-world event not mentioned in the text, the model cannot guess it. Additionally, this study focused exclusively on English Wikipedia; future work is needed to see how well this generalizes to the other 300+ languages Wikipedia supports. There is also the question of community norms—while synthetic data is cleaner, it might miss out on specific community-driven shorthand or "wiki-culture" nuances that human editors use.
Summary updated