Aligning Large Language Model Behavior with Human Citation Preferences

Kenichiro Ando, Tatsuya HaradaSubmission to ICLR 2026 · Rejected

TL;DR

While we’ve spent years perfecting how LLMs retrieve citations, we’ve ignored the question of when they should use them. This paper reveals a significant gap between what humans want cited (like numbers and names) and what LLMs actually cite (often just mimicking Wikipedia's "citation needed" tags), but demonstrates that Direct Preference Optimization (DPO) can successfully retrain models to follow human intuition.

What Problem Does This Paper Address?

In the world of AI-generated content, citations are the currency of trust. However, there is a "Goldilocks" problem: too few citations make a model seem untrustworthy, while too many lead to "information overload," where the user is buried under a mountain of links that hinder readability.

Most research to date has focused on the retrieval aspect—finding the right document to support a claim. This paper tackles the selection aspect: cite-worthiness. The authors argue that LLMs currently decide what to cite based on patterns in their training data rather than an understanding of what a human user actually needs to verify. If we can’t align LLM citation behavior with human preferences, we risk building "agentic" systems that either fail to justify critical claims or clutter their outputs with unnecessary references.

Methods

The researchers took a systematic approach to map out the "cite-worthiness" landscape:

  1. Taxonomy Creation: They identified eight categories of content that might trigger a citation (e.g., Medical Content, Jargon, Vague claims, and "Missing Information").
  2. Pairwise Evaluation: Instead of asking annotators a simple yes/no question, they presented pairs of sentences from different categories and asked, "Which of these two more urgently requires a citation?" This forced a ranking that reveals fine-grained human preferences.
  3. Model Benchmarking: They tested 11 prominent LLMs (including GPT-5, Claude Sonnet 4, and Llama 3) to see how their choices compared to the human baseline.
  4. Alignment Training: Finally, they attempted to "fix" the models using two common techniques: standard Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO).

Datasets

The study utilized WikiSQE, a large-scale collection of 3.4 million Wikipedia sentences that include "inline templates" (tags like [citation needed] or [vague]).

From this, they constructed a new, high-quality human-annotated dataset:

  • 6,000 sentences sampled across the eight categories.
  • 2,596 validated pairwise comparisons after rigorous quality control.
  • The dataset includes labels for human preferences vs. the preferences of 11 different LLMs.

Key Insights and Ideas

  • The Wikipedia Echo Chamber: LLMs are heavily biased by their training data. Models were up to 27.4% more likely than humans to demand a citation for text explicitly marked "citation needed" on Wikipedia. They aren't "thinking" about verifiability; they are memorizing where Wikipedia editors usually put tags.
  • The "Human" Blind Spot: Humans have a high demand for citations regarding numeric data and personal names. LLMs systematically under-selected these, with some models being over 20% less likely than humans to flag a name or a statistic as needing a source.
  • Medical Consensus: Interestingly, both humans and the strongest LLMs agreed that Medical Content is the highest priority for citations. This suggests that safety-alignment during model training has already successfully instilled some "caution" regarding health claims.
  • DPO vs. Fine-Tuning: The study found that standard fine-tuning actually decreased performance, likely because it tried to force "knowledge" the model didn't have. However, DPO (which trains on preferences directly) was highly effective, especially for smaller models like Llama 1B, which saw an 11.8% improvement in alignment.

Key Results

  • Low Initial Alignment: Current state-of-the-art models only agree with human citation preferences about 60% of the time.
  • Parameter Scale Effect: Larger models (like Llama 70B or GPT-5) generally align better with humans than tiny models (Llama 1B), but even the best models still suffer from the "Wikipedia bias."
  • Quantifiable Divergence: Models like Mistral Small under-selected numeric sentences by 22.6% relative to humans.
  • Successful Calibration: By using DPO, the researchers achieved a mean improvement of 5.76% across models, proving that cite-worthiness is a behavior that can be explicitly controlled and calibrated.

Limitations and Future Work

The authors acknowledge several constraints:

  • Domain & Language: The study is limited to English-language Wikipedia content. Citation needs in legal, creative, or scientific writing may differ significantly.
  • User Outcome: While the models now match human labels, the study doesn't measure if this actually improves user trust or task success in a real-world application.
  • Causality: The research identifies that models diverge from humans, but more work is needed to prove exactly why (e.g., pinpointing specific parts of the pre-training corpus).

Future work will likely involve integrating these "cite-worthiness" predictors directly into RAG pipelines to create smarter, cleaner, and more trustworthy AI agents.

Summary updated

Abstract

Most services built on powerful large-scale language models (LLMs) add citations to their output to enhance credibility. Recent research has paid increasing attention to the question of what reference documents to link to outputs. However, how LLMs recognize cite-worthiness and how this process should be controlled remains insufficiently explored. In this study, we focus on what kinds of content LLMs currently tend to cite and how well that behavior aligns with human preferences. We construct a dataset to characterize the relationship between human citation preferences and LLM behavior. Web-derived texts are categorized into eight citation-motivation types, and pairwise citation preferences are exhaustively evaluated across all type combinations to capture fine-grained contrasts. Our results show that humans most frequently seek citations for medical text, and stronger models display a similar tendency. We also find that current models are as much as 27% more likely than humans to add citations to text that is explicitly marked as needing citations on sources such as Wikipedia, and this overemphasis reduces alignment accuracy. Conversely, models systematically underselect numeric sentences (by -22.6% relative to humans) and sentences containing personal names (by -20.1%), categories for which humans typically demand citations. Furthermore, experiments with fine-tuning and Direct Preference Optimization (DPO) demonstrate that model behavior can be calibrated to better match human citation preferences. We expect this study to provide a foundation for more fine-grained investigations into LLM citation preferences. Our dataset and code will be released upon publication.
BibTeX
@misc{ando2026aligning,
  author = {Kenichiro Ando and Tatsuya Harada},
  title = {{Aligning Large Language Model Behavior with Human Citation Preferences}},
  year = {2026},
  eprint = {2602.05205},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.05205},
}
Figure