The Cooperation Databank: Machine-Readable Science Accelerates Research Synthesis

Giuliana Spadaro, Ilaria Tiddi, Simon Columbus, Shuxian Jin, Annette ten Teije, Daniel BallietPerspectives 2022 · DOI

TL;DR

This paper introduces the Cooperation Databank (CoDa), a novel machine-readable database of 2,636 studies on human cooperation. CoDa leverages ontologies and semantic modeling to enable on-demand meta-analyses and advanced research synthesis, aiming to accelerate scientific progress in the field.

What Problem Does This Paper Address?

The sheer volume of scientific publications makes it increasingly difficult for researchers to keep up with the literature, synthesize findings, and conduct comprehensive meta-analyses. This challenge is exacerbated by the fact that research results are often reported in unstandardized formats like PDFs, requiring extensive manual effort for data extraction and comparison. Furthermore, the reproducibility crisis highlights the need for more robust and transparent methods for aggregating and analyzing research findings. Existing meta-analysis tools can be time-consuming and inaccessible to non-experts.

Methods

The core of CoDa's methodology lies in its application of knowledge representation techniques, specifically ontology engineering, to create a semantically enhanced databank.

  1. Systematic Search: A comprehensive search was conducted for studies on human cooperation published in English, Chinese, and Japanese up to 2017.
  2. Ontology Development: An ontology of human cooperation studies was developed. This ontology defines concepts and their relationships within the domain, creating a structured model for representing research findings. It includes both domain-independent bibliographic information and domain-specific knowledge about cooperation.
  3. Expert Annotation: Studies were meticulously annotated by domain experts along 312 variables, covering study metadata, sample characteristics, and quantitative results (including 13,959 effect sizes). This annotation process was designed to minimize human error through training and interrater agreement checks.
  4. Data Conversion and Publication: The annotated data was converted into a structured, graph-shaped dataset adhering to FAIR principles (Findability, Accessibility, Interoperability, Reusability). This dataset is publicly accessible and queryable.
  5. Research Platform Development: An R Shiny web application was built to provide a user-friendly interface for exploring the databank. This platform enables users to perform meta-analyses, metaregressions, publication bias assessments, and statistical power analyses on demand.

Datasets

The primary dataset introduced and utilized in this paper is the Cooperation Databank (CoDa).

  • Content: CoDa contains 2,636 studies on human cooperation.
  • Timeframe: Studies range from 1958 to 2017.
  • Scope: Covers research conducted in 78 societies, involving 356,283 participants.
  • Annotation: Studies are annotated with 312 variables, including 13,959 quantitative results (effect sizes).
  • Format: Machine-readable and semantically enhanced through an ontology.

This dataset is novel in its comprehensive nature, machine-readability, and the integration of a formal ontology for representing cooperation research.

Key Insights and Ideas

  • Machine-Readable Science is the Future: The paper strongly advocates for publishing research in standardized, machine-readable formats. This is presented not just as a convenience but as a fundamental shift that will enable automated analysis and accelerate scientific discovery.
  • Ontologies as the Backbone of Synthesis: The development and application of a domain-specific ontology are crucial. It provides the structured framework necessary to represent complex relationships between variables and study outcomes, allowing for sophisticated reasoning and analysis by machines.
  • On-Demand Meta-Analysis is Achievable: CoDa demonstrates that it's possible to move beyond lengthy, manual meta-analysis projects to a system where researchers can perform these analyses dynamically based on their specific research questions.
  • Bridging the Gap Between Experimental and Correlational Data: The ontology is designed to represent relationships between both correlational and experimental study results, a significant step towards unifying diverse research methodologies.
  • Addressing the Reproducibility and Transparency Challenge: By making data and analysis methods explicit and machine-readable, CoDa inherently promotes transparency and facilitates the reproduction of research syntheses.

Key Results

  • CoDa Databank: A comprehensive databank of 2,636 studies on human cooperation, encompassing data from 78 societies and over 356,000 participants, has been created.
  • Rich Annotation: Studies are annotated with 312 variables, yielding 13,959 effect sizes, providing a granular view of the research landscape.
  • Functional Research Platform: The CoDa platform enables users to:
  • Perform on-demand meta-analyses and metaregressions.
  • Estimate and adjust for publication bias.
  • Conduct statistical power analyses for future studies.
  • Explore the research domain through interactive visualizations of the ontology and citation networks.
  • Interrater Agreement: High interrater agreement (Krippendorff's α above .70) was achieved for most variables during the annotation process, indicating the reliability of the data.

Limitations and Future Work

  • Geographic Bias: While efforts were made to include non-Western studies (Chinese and Japanese), the databank still has a significant representation from Western, educated, industrialized, rich, and democratic (WEIRD) societies (41% from the US).
  • Risk of Bias Assessment: CoDa currently does not provide an explicit estimate of the risk of bias within individual studies, which is a common challenge in meta-analysis.
  • Sustainability of Expert Annotation: The reliance on trained domain experts for data extraction presents a challenge for long-term sustainability.
  • Future Work:
    • Automated Annotation: Developing machine learning and natural language processing techniques to automate tasks like study search and data extraction.
  • Expanding Scope: Extending the ontology and databank to include other topics and disciplines beyond human cooperation.
  • Community Contribution: Implementing mechanisms for researchers worldwide to submit their studies for inclusion.
  • Risk of Bias Integration: Developing standards and tools to assess the risk of bias within cooperation studies.
  • Living Reviews: Further developing the platform to support automatically updated "living reviews" of research findings.

Summary updated

Abstract

Publishing studies using standardized, machine-readable formats will enable machines to perform meta-analyses on demand. To build a semantically enhanced technology that embodies these functions, we developed the Cooperation Databank (CoDa)-a databank that contains 2,636 studies on human cooperation (1958-2017) conducted in 78 societies involving 356,283 participants. Experts annotated these studies along 312 variables, including the quantitative results (13,959 effects). We designed an ontology that defines and relates concepts in cooperation research and that can represent the relationships between results of correlational and experimental studies. We have created a research platform that, given the data set, enables users to retrieve studies that test the relation of variables with cooperation, visualize these study results, and perform (a) meta-analyses, (b) metaregressions, (c) estimates of publication bias, and (d) statistical power analyses for future studies. We leveraged the data set with visualization tools that allow users to explore the ontology of concepts in cooperation research and to plot a citation network of the history of studies. CoDa offers a vision of how publishing studies in a machine-readable format can establish institutions and tools that improve scientific practices and knowledge.
BibTeX
@inproceedings{spadaro2022cooperation,
  author = {Giuliana Spadaro and Ilaria Tiddi and Simon Columbus and Shuxian Jin and Annette ten Teije and Daniel Balliet and Daniel Balliet},
  title = {{The Cooperation Databank: Machine-Readable Science Accelerates Research Synthesis}},
  booktitle = {Perspectives},
  year = {2022},
  doi = {10.1177/17456916211053319},
}
Figure