PersonalSum: A User-Subjective Guided Personalized Summarization Dataset for Large Language Models

Lemei Zhang, Peng Liu, Marcus Tiedemann Oekland Henriksboe, Even W. Lauvrak, Jon Atle Gulla, Heri RamampiaroNeurIPS 2024

Introduction

Researchers have introduced PersonalSum, the first high-quality, human-annotated dataset designed to help Large Language Models (LLMs) move beyond "one-size-fits-all" summaries. By capturing individual user profiles and subjective interests across Norwegian news articles, the dataset reveals that personalized summarization requires more than just tracking keywords—it demands an understanding of a user's preferred narrative "plot" and article structure.

The Problem: Beyond Generic Summaries

While LLMs like GPT-4 are now capable of producing generic summaries that rival professional journalists, these summaries often fail to meet the specific needs of individual readers. One person might read a news story to find specific financial data (selective reading), while another just wants the "big picture" (scanning).

The primary bottleneck in solving this is a lack of high-quality data. Most existing summarization datasets are either generic (providing only one summary per article) or controllable (generating summaries based on a specific entity or length constraint). They rarely reflect the messy, subjective reality of what an actual human finds interesting. Furthermore, existing personalized datasets often rely on machine-generated "pseudo-data" rather than genuine human intuition.

Methodology

The authors developed a rigorous three-stage pipeline to build the PersonalSum dataset:

  1. Generic Baseline: They selected 465 Norwegian news articles and used GPT-4 to generate generic summaries. These were then manually refined by native Norwegian speakers to ensure 100% factual consistency.
  2. Human Personalization: Annotators were recruited via Amazon Mechanical Turk. Unlike typical tasks, these users didn't just write a summary; they provided a user profile, highlighted the specific source sentences that interested them, and answered comprehension questions to ensure high-quality engagement.
  3. LLM-in-the-Loop Quality Control: To filter out low-effort annotations, the team used GPT-3.5-Turbo as a "judge" to score the relevance between the human summaries and their highlighted sources.

The PersonalSum Dataset

The core contribution of this work is the PersonalSum dataset, featuring 1,816 personalized summaries across 441 Norwegian news articles, complete with user profiles and source sentence mapping.

Dataset demographics and annotation statistics

To address "feature sparsity," the researchers also created a Topic-centric PersonalSum subset. This groups articles that share the same entities, allowing models to better learn a user's specific preferences for certain topics over multiple examples. Compared to standard benchmarks like CNN/DM and PENS, PersonalSum stands out by including explicit user profiles and direct source-to-summary mapping.

Key Insights and Ideas

Benchmarking several LLMs (GPT-3.5-Turbo, Gemini 1.0 Pro, and NorwAI-Mixtral) using few-shot in-context learning revealed several critical insights about how models handle personalization:

  • The "Lead" Bias: Generic summaries (and the LLMs that produce them) are heavily biased toward the beginning of an article. In contrast, PersonalSum shows that human interests are distributed much more evenly throughout a text.
Distribution of source sentences and summary lengths, highlighting the machine 'lead bias'
  • More Data $\neq$ Better Results: Surprisingly, the researchers found that 5-shot prompting often outperformed 10-shot prompting. Providing too much historical user data actually introduced "noise," causing the models to lose focus on the current article.
  • Beyond Keywords: While "Entities" (names, places, topics) are a major factor in personalization, they aren't everything. Incorporating the "Plot" (the cause and result of events) and "Position" (the user's structural preference) significantly improved the quality of the generated summaries.
  • The Subjectivity Gap: The study confirms that personalized summarization remains a significant challenge. Even with sophisticated prompts, LLMs struggle to capture the subtle, subjective nuances that make a summary feel "right" to a specific individual.

Key Results

The experiments showed that incorporating personalized signals consistently improved performance over generic summaries:

  • Quantitative Improvements: Using "All" factors (Entity + Plot + Position) in a 5-shot setting typically yielded the highest ROUGE-1 scores. For example, GPT-3.5-Turbo saw its ROUGE-1 score climb from 37.61 (Generic) to 39.96 (All factors) on the Topic-centric dataset.
  • Qualitative Success: Human evaluators (native Norwegian speakers) rated the personalized outputs and found high agreement (Fleiss’ kappa > 0.8) that summaries incorporating user-specific factors were more coherent and aligned better with user needs than generic ones.
  • Model Comparisons: GPT-3.5-Turbo generally outperformed Gemini and Mixtral in following complex personalized instructions, though all models occasionally struggled with including "excessive detail" in their summaries.

Limitations and Future Work

The authors acknowledge several hurdles in the current iteration of the dataset. With 1,816 summaries, the dataset is high-quality but relatively small for training massive models from scratch. Additionally, the dataset is currently limited to Norwegian, which may affect how well the results generalize to English-centric LLMs (the authors noted Llama-3 struggled with the Norwegian prompts). Finally, annotators were assigned articles rather than choosing them based on genuine personal interest, which might slightly dilute the "subjectivity" of the data.

Future work will likely focus on scaling the dataset and exploring how to better balance user history to prevent the "noise" issues observed in higher-shot prompting scenarios.

Summary updated

BibTeX
@inproceedings{zhang2024personalsum,
  author = {Lemei Zhang and Peng Liu and Marcus Tiedemann Oekland Henriksboe and Even W. Lauvrak and Jon Atle Gulla and Heri Ramampiaro},
  title = {{PersonalSum: A User-Subjective Guided Personalized Summarization Dataset for Large Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
}
Figure