In the ideal version of science, a citation is a formal acknowledgment of intellectual debt—a sign that one piece of work provided the foundation, the data, or the method for another. If this were true, citation counts would be a perfect proxy for scientific quality.
However, a growing body of research suggests that citation metrics have largely decoupled from the underlying technical rigor of the work they measure. Instead of tracking "truth" or "quality," our current metrics often track social phenomena: prestige, discoverability, and the mathematical probability of being noticed.
The Feedback Loop of Lazy Citing
The most fundamental reason citation counts fail as a quality metric is that they are often generated through a process of "lazy citing." By analyzing identical misprints in bibliographies, M. V. Simkin discovered in An introduction to the theory of citing that roughly 80% of citations are copied from other reference lists rather than the original source.
This creates a self-reinforcing feedback loop. A paper becomes highly cited not necessarily because it is the most rigorous, but because it was cited early and often, making it easier for subsequent authors to find and copy. As M. V. Simkin argues, scientific "greatness" may often be a product of mathematical probability—a "bestseller" effect—rather than pure intellectual merit.
Popularity is Not Prestige
If citations are a measure of popularity, they are distinct from what Ying Ding 0001 and Blaise Cronin call "prestige." In Popular and/or Prestigious? Measures of Scholarly Esteem, they distinguish between how many people cite you (popularity) and who is citing you (prestige).
This distinction is critical because raw popularity is a poor predictor of long-term academic honors. High-impact journals often prioritize papers that generate immediate "buzz" or high citation volume, but these papers do not always correlate with the methodological rigor required for reproducibility. For instance, Joe D. Menke and colleagues developed the Rigor and Transparency Index (RTI) to score papers on technical details like blinding and randomization. Their analysis of 1.6 million papers found zero correlation between a journal’s Impact Factor and the technical rigor of its articles.
The Failure of High-Impact Gatekeeping
We often assume that the peer-review process at "top-tier" venues acts as a filter for quality. However, evidence from the machine learning community suggests otherwise. David Tran and Micah Goldblum found in Analyzing the Machine Learning Conference Review Process that review scores at conferences like ICLR show a very low correlation with eventual citation impact.
Even more concerning is that high-impact publication does not guarantee actionable evidence. Noah Haber and colleagues conducted a systematic review of COVID-19 health policy evaluations and found that 89% of studies failed basic methodological standards for causal inference, despite being published in prestigious journals and heavily cited. In these cases, the "prestige" of the venue and the "popularity" of the topic masked fundamental flaws in the research design.
The Double Status Effect
Why does the community continue to cite flawed but high-status work? Eamon Duede and Misha Teplitskiy identify a "double status effect": highly cited papers aren't just cited more often; they are perceived as more authoritative, leading authors to read them more closely and cite them more "meaningfully."
Conversely, work from less prestigious institutions or authors is often overlooked, regardless of its quality. This social bias is even present in automated systems; Patricia Maes found that Large Language Models (LLMs) evaluating economics papers exhibited significant "prestige bias," favoring submissions from elite institutions over the actual content of the research.
Moving Toward Content-Aware Metrics
To fix the "broken compass" of bibliometrics, researchers are moving toward metrics that analyze the context and substance of citations rather than just the count.
- Sentiment and Support: Domenic Rosati and Joshua M. Nicholson proposed the Scite Index, which uses machine learning to distinguish between citations that support, mention, or contrast a paper’s findings. They found that a journal's impact factor is almost entirely uncorrelated with the degree to which its findings are actually supported by subsequent studies.
- Surprise and Innovation: Feng Shi and James Evans argue in Science and Technology Advance through Surprise that we should value "surprising" combinations of ideas. While 95% of citations follow predictable patterns, the 5% of "improbable" combinations are the ones that actually predict Nobel-level breakthroughs.
- Intellectual Lineage: Tools like PST-Bench, developed by Fanjin Zhang and colleagues, attempt to trace the "direct sources" of inspiration. They found that the most-cited references in a paper are the actual source of its core ideas less than 10% of the time.
By shifting our focus from raw counts to the quality of the "intellectual signal," the research community can begin to bridge the gap between what is popular and what is true.