Data and Code Availability in Political Science Publications from 1995 to 2022

Carlisle Rainey, H.s.j. Roe, Qing Wang, Hao Zhou

TL;DR

While political science has championed "open science" for decades, only about 31% of quantitative research articles published in 2022 actually point to a reproduction archive. While this represents a significant jump from just 12% in 2014, the majority of the field’s research remains computationally unverifiable, with a massive "prestige gap" between top-tier journals and the rest of the discipline.

What Problem Does This Paper Address?

In the "hard" sciences and software engineering, reproducibility is a baseline requirement. In social sciences, however, the "reproducibility crisis" has highlighted a troubling gap: many published results cannot be verified because the original data and code are missing.

While previous studies looked at "top-tier" journals (which often have strict sharing mandates), we didn't have a clear picture of the entire field. The authors set out to document how the availability of "reproduction archives"—the specific data and code needed to recreate a paper's tables and figures—has evolved across the broad spectrum of political science from 1995 to 2022.

Methods

The researchers employed a rigorous three-step sampling and coding process:

  1. Sampling: They identified 224 English-language journals in the Social Science Citation Index (SSCI) under "Political Science" and "International Relations." Using Crossref’s API, they pulled over 560,000 DOIs and took a random sample of 5,000, plus an over-sample of 1,000 from 2022 to ensure contemporary accuracy.
  2. Filtering: A team of four coders manually reviewed the DOIs to identify "quantitative research articles"—those with data that could be shared. This left a final sample of 1,413 articles.
  3. Classification: Each article was coded into seven categories, ranging from "not mentioned" to "linked and still available."
  4. Modeling: To account for noise in year-to-year sampling, they used a monotonic regression model (via the brms package in R) to estimate the underlying trend of increasing availability over time.

Datasets

The primary "dataset" for this study is the curated collection of 1,413 quantitative research articles sampled from the SSCI. The authors have practiced what they preach: the data and code used to produce this specific paper are publicly available on Dataverse.

Key Insights and Ideas

  • The "Prestige Gap": There is a stark divide based on journal ranking. In 2022, 74% of articles in the top quartile of journals (like American Political Science Review) provided reproduction archives. In the bottom two quartiles, that number plummeted to just 13%.
  • The APSA Effect: Journals affiliated with the American Political Science Association (APSA), which has a formal code of ethics regarding data sharing, perform significantly better (57% availability) than non-APSA journals (27%).
  • Infrastructure Works: The "broken link" problem is surprisingly small. Only about 2% of articles had links that no longer worked. This suggests that the shift toward permanent repositories like Dataverse and OSF (Open Science Framework) is successfully preventing "link rot."
  • The Definition of "Available": The authors distinguish between "replication" (getting similar results with new data) and "computational reproducibility" (getting the same results with the original data/code). They argue that without the latter, the former is nearly impossible to evaluate.

Key Results

  • 1995–2002: Essentially 0% of sampled articles shared data and code. The first instance in their sample didn't appear until 2003.
  • 2014: Around the time of major disciplinary symposia on transparency (DA-RT), availability sat at 12%.
  • 2022: Availability reached 31%. While this is a "tripling" of the 2014 rate, it still means 69% of quantitative political science research is not publicly reproducible.
  • The "Not Mentioned" Problem: Even in 2022, 57% of articles make no mention of data or code availability whatsoever.

Limitations and Future Work

  • Quantity vs. Quality: The authors only checked if a reproduction archive was pointed to and accessible. They did not attempt to actually run the code to see if it produced the correct results. An archive could be "available" but contain broken code or incomplete data.
  • Policy Linkage: While the authors suspect journal requirements drive these trends, they couldn't perfectly map historical journal policies to every paper in the sample due to the long lag times in publishing.
  • Qualitative Data: This study focused strictly on quantitative work. The authors acknowledge that sharing qualitative data (interviews, archives) presents different ethical and practical challenges that require separate study.

Summary updated

Abstract

In this paper, we assess the availability of reproduction archives in political science. By “reproduction archive,” we mean the data and code supporting quantitative research articles that allows others to reproduce the computations described in the published paper. We collect a random sample of quantitative research articles published in political science from 1995 to 2022. We find that—even in 2022—most quantitative research articles do not point a reproduction archive. However, practices are improving. In 2014, when the DA-RT symposium was published in PS, about 12% of quantitative research articles point to the data and code. Eight years later, in 2022, that has increased to 31%. This underscores a massive shift in norms, requirements, and infrastructure. Still, only a minority of articles share the supporting data and code. In 2014, Lupia and Alter wrote: “Today, information on the data production and analytic decisions that underlie many published works in political science is unavailable.” They could write the same today; much work remains to be done.
BibTeX
@misc{rainey2024data,
  author = {Carlisle Rainey and H.S.J. Roe and Qing Wang and Hao Zhou},
  title = {{Data and Code Availability in Political Science Publications from 1995 to 2022}},
  year = {2024},
  doi = {10.31235/osf.io/a5yxe},
}
Figure