Can LLMs replace Neil deGrasse Tyson? Evaluating the Reliability of LLMs as Science Communicators

Prasoon Bajpai, Niladri Chatterjee, Subhabrata Dutta, Tanmoy Chakraborty

Introduction

As large language models (LLMs) are increasingly integrated into academic and scientific workflows, their role as "science communicators" has come under scrutiny. While these models can synthesize vast amounts of information, a critical question remains: can they be trusted to communicate complex scientific concepts accurately, or do they merely provide authoritative-sounding misinformation? Researchers from IIT Delhi have addressed this by introducing SCiPS-QA, a rigorous new benchmark designed to evaluate whether LLMs possess the nuanced understanding and "epistemic humility" required for reliable science communication.

The Problem: Authoritative Misinformation

Existing STEM benchmarks, such as MMLU or GSM8K, primarily focus on factual recall or standard multi-step mathematical problems. However, they often fail to test for answerability—the ability of a model to recognize when a scientific question is an "open problem" with no definitive answer.

A true science communicator must be self-aware enough to admit when a question is beyond current scientific consensus. If an LLM provides a wrong answer with extreme confidence and a persuasive explanation, it does more than fail a test; it actively spreads scientific misconceptions. The IIT Delhi study explores this "persuasion trap," where the stylistic quality of an AI's response masks its factual inaccuracy.

Methodology

The researchers developed a comprehensive evaluation suite to move beyond simple accuracy metrics:

  • Multi-Model Benchmarking: The study evaluated 16 models, including the proprietary GPT family and open-access models from the Llama-2, Llama-3, and Mistral families.
  • Hallucination Quantification: Using SelfCheckGPT, the team generated multiple "stochastic" (randomized) responses to the same prompt. High variance or contradiction between these responses indicated a high probability of hallucination. This was measured using BERTScore (semantic similarity), NLI (logical entailment), and prompt-based judging.
  • NLG Evaluation: The study measured the "Convince-factor" (persuasiveness of reasoning), "Fact-check" (factual accuracy), and "Information Mismatch."
  • Human-in-the-loop: Graduate-level subject experts rated the "convincingness" of GPT-4’s reasoning to determine if experts could be deceived by incorrect but well-structured AI explanations.

The SCiPS-QA Dataset

The SCiPS-QA (Specially Challenging Problems in Science – Question Answering) dataset consists of 742 complex Yes/No queries across Physics, Chemistry, and Mathematics. It is divided into two categories:

  1. Closed Questions (510): Problems with definitive answers supported by existing scientific literature.
  2. Open Questions (232): Niche, scientifically unanswerable, or currently debated problems (e.g., "Is Tarski's exponential function problem decidable?").

The study found that GPT-4 Turbo performs significantly worse on SCiPS-QA than on popular datasets like MMLU-Pro, indicating that SCiPS-QA represents a much higher bar for scientific reasoning.

Key Insights

The Persuasion Trap

One of the most alarming findings is that GPT-4 Turbo’s authoritative tone is a double-edged sword. Its reasoning is so convincing that human experts were frequently misled by incorrect explanations. Furthermore, the study debunked the "self-correction myth": when GPT-4 was asked to verify its own reasoning, it often gave its own incorrect answers a perfect "convincingness" score.

Llama-3-70B: A Strong Contender

In a surprising result, the open-source Llama-3-70B model achieved the highest Main Response Accuracy (0.693), surpassing GPT-4 Turbo (0.646). While GPT-4 Turbo showed the lowest variation in responses (indicating high consistency), it was often consistently wrong.

The Abstention Gap

Most models failed the "I don't know" test. Instead of identifying an open scientific problem as unanswerable, they typically picked a side and hallucinated a justification. Llama-3-70B was the most successful at identifying unanswerable questions (0.582), whereas Llama-2 models struggled significantly, often failing to follow the requested format entirely.

Limitations and Future Directions

The researchers noted several areas for future development:

  • Golden Reasoning: The dataset currently lacks human-verified "perfect" reasoning passages to serve as a ground truth for explanations.
  • Topic Diversity: While broad, the dataset is currently weighted toward specific sub-fields like Quantum Mechanics and requires more diversity.
  • Knowledge Cutoff: Some errors may stem from training data cutoffs rather than failures in logic.
  • Human Evaluation Scale: Larger-scale human studies are needed to fully quantify the "deception effect" across different scientific disciplines.

Conclusion

While LLMs are impressive tools, they are not yet ready to replace human science communicators. Their lack of epistemic humility—the ability to recognize the limits of their own knowledge—remains a critical barrier to trust. For now, the role of the reliable science communicator remains firmly in human hands.

Summary updated

Abstract

Large Language Models (LLMs) and AI assistants driven by these models are experiencing exponential growth in usage among both expert and amateur users. In this work, we focus on evaluating the reliability of current LLMs as science communicators. Unlike existing benchmarks, our approach emphasizes assessing these models on scientific questionanswering tasks that require a nuanced understanding and awareness of answerability. We introduce a novel dataset, SCiPS-QA, comprising 742 Yes/No queries embedded in complex scientific concepts, along with a benchmarking suite that evaluates LLMs for correctness and consistency across various criteria. We benchmark three proprietary LLMs from the OpenAI GPT family and 13 open-access LLMs from the Meta Llama-2, Llama-3, and Mistral families. While most open-access models significantly underperform compared to GPT-4 Turbo, our experiments identify Llama-3-70B as a strong competitor, often surpassing GPT-4 Turbo in various evaluation aspects. We also find that even the GPT models exhibit a general incompetence in reliably verifying LLM responses. Moreover, we observe an alarming trend where human evaluators are deceived by incorrect responses from GPT-4 Turbo.
BibTeX
@inproceedings{bajpai2024can,
  author = {Prasoon Bajpai and Niladri Chatterjee and Subhabrata Dutta and Tanmoy Chakraborty},
  title = {{Can LLMs replace Neil deGrasse Tyson? Evaluating the Reliability of LLMs as Science Communicators}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.889},
}
Figure