Trust in science, trust in ChatGPT? How Germans think about generative AI as a source in science communication

Mike S. Schäfer, Bastian Kremer, Niels G. Mede, Liliann FischerJournal of Science Communication 2024 · DOI

TL;DR

This study investigates how Germans perceive and trust generative AI (GenAI) tools like ChatGPT as sources of science-related information. It finds that while Germans appreciate GenAI's ability to simplify complex scientific content, they remain largely skeptical and do not strongly trust it. Trust in GenAI is significantly influenced by general trust in science, which in turn is shaped by scientific knowledge and the perception of science improving quality of life.

What Problem Does This Paper Address?

The rapid rise of generative AI (GenAI) tools, such as ChatGPT, has positioned them as increasingly important sources of information across various domains, including science communication. However, there's a significant knowledge gap regarding public trust in these AI tools as sources for scientific content. This paper addresses this by exploring how Germans, a population with generally high trust in science, perceive and trust GenAI in the context of science communication. It aims to understand the factors that drive this trust, or lack thereof, in an emerging and potentially transformative technology.

Methods

The study employs a quantitative approach using structural equation modeling (SEM) to analyze data from the German Science Barometer 2023. The authors adapt a conceptual model developed by Roberts et al. (2013) to investigate the relationships between sociodemographic variables, attitudes towards science, and trust in GenAI as a source for science communication. The SEM allows for the examination of direct and indirect effects of various factors on trust.

Datasets

The study utilizes data from the German Science Barometer 2023. This is a nationally representative telephone survey of the German population aged 14 and older, conducted in August 2023. The survey included questions specifically addressing attitudes towards and trust in GenAI (referred to as "programs such as ChatGPT") as a source of science-related information.

Key Insights and Ideas

  • Skepticism towards GenAI in science communication is prevalent: Despite appreciating GenAI's ability to simplify complex scientific topics, a significant portion of Germans express skepticism and do not strongly trust it as a source for scientific information.
  • Trust in science is a crucial gateway: General trust in science acts as a strong predictor for trust in GenAI for science communication. This suggests that a foundational belief in the scientific enterprise is a prerequisite for accepting AI as a reliable scientific information source.
  • Knowledge and perceived benefits drive trust in science: Trust in science itself is largely driven by individuals' self-perceived knowledge about science and their belief that science improves their quality of life.
  • Demographics play a role, but with nuances: While sociodemographic factors like age, education, and income influence attitudes towards science and GenAI, their impact is complex and not always straightforward, with younger individuals showing higher trust in GenAI.
  • GenAI's "human-like" interaction is not a primary trust driver: Unlike some prior research suggesting human-like interaction enhances trust in AI, German citizens in this study do not appear to prioritize this aspect when evaluating GenAI for science communication.

Key Results

  • Trust in GenAI: Only 17% of respondents expressed trust or complete trust in programs like ChatGPT for reproducing scientific content. A substantial 46% did not trust it, and 36% remained undecided.
  • Perceived positive aspects of GenAI: 53% agreed that GenAI's ability to explain complex scientific content in a simplified way is positive. 50% found its ability to provide examples and answer questions helpful.
  • Perceived negative aspects of GenAI: A clear majority (63%) agreed with concerns about GenAI's potential to disseminate misinformation, its lack of transparency, and inadequate source verification.
  • SEM findings:
    • General trust in science was a significant and positive predictor of trust in GenAI for science communication.
  • Attitudes towards science's impact on quality of life (perceptions of risks and benefits) were strong predictors of general trust in science.
  • Self-perceived knowledge showed a slight positive effect on general trust in science.
  • Personal attachment to science did not show a significant effect on trust in this model.
  • Age was a notable factor, with younger respondents (14-29) showing significantly higher trust in GenAI (46%) compared to older age groups (10-17%).

Limitations and Future Work

The authors acknowledge several limitations:

  • Discrepancies with prior models: Differences observed compared to Roberts et al. (2013) highlight the complexity of trust in AI and the need for further exploration of underlying factors.
  • GenAI tool specificity: The study focused on "programs such as ChatGPT" and did not differentiate between various GenAI tools, which may have different trustworthiness profiles due to varying transparency.
  • Conceptual model expansion: Future research could incorporate factors from Technology Acceptance Models, such as perceived ease of use and user familiarity.
  • Data limitations: As a secondary analysis of existing survey data, the study was constrained by the available questions. Future studies could employ more detailed measures.

The authors suggest that future research should continue to explore these factors, potentially examining how the influence of variables like education and age might differ for other technologies. They also emphasize the need for ongoing discussions about the use of GenAI in science communication, its applications, limits, and whether trust should be actively promoted. Practical implications include informing science communicators on how to approach and use GenAI, and improving users' GenAI literacy.

Summary updated

Abstract

Generative AI like ChatGPT has been diagnosed to fundamentally impact different realms of life. This includes science communication, where GenAI tools are becoming important sources of science-related content for many people. This raises the question of whether people trust GenAI as a source in this field, a question that has not been answered sufficiently yet. Adapting a model developed by Roberts et al. [2013] and utilizing survey data from the German Science Barometer 2023, we find that Germans are rather sceptical about and do not strongly trust GenAI in science communication. Structural equation modelling shows that respondents' trust in GenAI as a source in science communication is driven strongly by their general trust in science, which is largely driven by their knowledge about science and the perception that science improves quality of life.
BibTeX
@article{schafer2024trust,
  author = {Mike S. Schäfer and Bastian Kremer and Niels G. Mede and Liliann Fischer},
  title = {{Trust in science, trust in ChatGPT? How Germans think about generative AI as a source in science communication}},
  journal = {Journal of Science Communication},
  year = {2024},
  doi = {10.22323/2.23090204},
}
Figure