Explain the Black Box for the Sake of Science: Revisiting the Scientific Method in the Era of Generative Artificial Intelligence

Gianmarco Mengaldo

Introduction

In recent years, artificial intelligence (AI) has achieved superhuman performance in complex scientific domains, from weather forecasting to protein folding. However, for the scientific community, a correct prediction is only a partial victory. The traditional scientific method is rooted in understanding the underlying principles—the "why" behind a phenomenon.

Currently, many high-performing AI models function as "black boxes." While their predictive power is undeniable, the specific patterns or data relationships they exploit often remain opaque. This creates a significant "knowledge gap": AI may have uncovered novel principles of nature that remain inaccessible to human understanding. Without robust interpretability, AI risks becoming a dead-end for scientific insight rather than a catalyst for discovery.

The XAI for Science Framework

To bridge this gap, this paper proposes a framework called Explainable AI for Science. This approach moves beyond using AI solely for prediction, repositioning it as a tool for active scientific discovery. By integrating interpretability methods into the classical inductive reasoning process, researchers can extract a "machine view" of a problem.

The framework relies on three core components:

  1. The Machine View: Utilizing interpretability tools to identify which data features the AI prioritizes when making a decision.
  2. The ARU Pillars: To ensure scientific utility, the "machine view" must satisfy three criteria: accuracy (faithfully representing the model), reproducibility (consistency across trials), and understandability (relatability to domain knowledge).
  3. Interpretability-Guided Explanations (IGEs): This is the synthesis stage where human scientists (potentially aided by Generative AI) compare the "machine view" with the "human view" (existing knowledge).
The proposed framework comparing human and AI workflows to identify scientific principles.

The paper distinguishes between post-hoc interpretability (explaining a pre-trained model using tools like SHAP or GradCAM) and ante-hoc interpretability (employing inherently transparent models like decision trees).

Case Studies: Validating the Workflow

The proposed workflow was tested using two distinct datasets, with Transformer neural networks serving as the predictive engines:

  • Medical Diagnostics (ECG Time-Series): In classifying heart pathologies like Right Bundle Branch Block (RBBB), the AI (via SHAP) identified the delayed and broadened R'-wave in the V1 lead as the primary diagnostic feature. This convergent view aligns with established medical principles, validating the AI's trustworthiness.
  • Climate Science (Spatio-temporal Weather Data): For forecasting extreme heatwaves in Indochina, the AI prioritized rising temperatures (T2M) and high-pressure anomalies (SLP) while correctly identifying low precipitation as a minor factor. This allows scientists to pinpoint the specific atmospheric "windows" the AI monitors.
The XAI for Science workflow applied to ECG signals and weather forecasting.

Discussion and Future Directions

A key insight of this framework is that divergence between human and machine views is not a failure, but a catalyst for inquiry. When an AI identifies a feature that contradicts or extends human knowledge, it signals a potential new scientific discovery.

Furthermore, the paper explores the role of Generative AI (GenAI) as a "scientist's assistant." Large Language Models (LLMs) can help bridge the gap by translating raw interpretability data into human-readable hypotheses, acting as a bridge between the "machine view" and the "human view."

However, challenges remain. AI is prone to finding correlations rather than causality, leading to potential "scientific hallucinations." Additionally, while we can see what data a model uses, understanding how it mathematically relates those features is still difficult. Future work may involve symbolic regression to transform these data patterns into explicit mathematical laws, fully integrating AI into the scientific method.

Summary updated

Abstract

The scientific method is the cornerstone of human progress across all branches of the natural and applied sciences, from understanding the human body to explaining how the universe works. The scientific method is based on identifying systematic rules or principles that describe the phenomenon of interest in a reproducible way that can be validated through experimental evidence. In the era of artificial intelligence (AI), there are discussions on how AI systems may discover new knowledge. We argue that human complex reasoning for scientific discovery remains of vital importance, at least before the advent of artificial general intelligence. Yet, AI can be leveraged for scientific discovery via explainable AI. More specifically, knowing what data AI systems deemed important to make decisions can be a point of contact with domain experts and scientists, that can lead to divergent or convergent views on a given scientific problem. Divergent views may spark further scientific investigations leading to new scientific knowledge.
BibTeX
@misc{mengaldo2024explain,
  author = {Gianmarco Mengaldo},
  title = {{Explain the Black Box for the Sake of Science: Revisiting the Scientific Method in the Era of Generative Artificial Intelligence}},
  year = {2024},
  eprint = {2406.10557},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.10557},
}
Figure