The Incomplete Bridge: How AI Research (Mis)Engages with Psychology

Han Jiang, Pengda Wang, Xiaoyuan Yi, Xing Xie, Ziang XiaoarXiv 2025 · DOI · arXiv

Introduction

As Large Language Models (LLMs) evolve into increasingly complex "black boxes," their behaviors often transcend simple algorithmic explanations. To understand how these systems "think," reason, and interact, AI researchers have increasingly turned to the field of psychology. This shift treats LLMs not just as software, but as subjects for experimental inquiry. However, a recent large-scale bibliometric analysis reveals that while the bridge between AI and psychology is being built rapidly, it remains structurally incomplete.

The Problem: A Consensus of Misreading

The core challenge lies in the "Incomplete Bridge" problem. Interdisciplinary work requires a shared vocabulary, yet AI researchers often adopt psychological terms—such as "attention," "Theory of Mind," or "reasoning"—without fully embracing their original theoretical nuances. This leads to a "consensus of misreading," where flawed interpretations are codified into AI benchmarks. When these benchmarks become the standard for evaluation, they risk eroding scientific credibility and producing skewed conclusions about the true capabilities of artificial intelligence.

Mapping the Interdisciplinary Landscape

To quantify this phenomenon, researchers conducted a "science-of-science" analysis of over 25,000 LLM-related articles published between 2023 and 2025. By filtering for papers that specifically cited psychology literature, they identified a core corpus of 1,006 AI papers referencing 2,544 psychology publications.

Using the SPECTER transformer model and K-means clustering, the study mapped these works into distinct research clusters.

Hierarchical clustering of LLM and Psychology research topics.

As shown in the figure above, LLM research topics were categorized into domains like Cognition & Reasoning, Human-Centric AI, and Model Adaptation. Psychology topics were similarly clustered into areas such as Social-Clinical, Psychometrics, and Neural Mechanisms.

The Citation Gap and the "GPT-4 Spark"

The data reveals a dramatic spike in citations to psychology research around March 2023, coinciding with the release of GPT-4. As models began to exhibit more "human-like" behaviors, the AI community felt an urgent need for psychological tools to measure them. However, this interest is highly selective.

Bipartite network mapping the connections between LLM research topics and psychology domains.

The bipartite network in the figure above illustrates the uneven distribution of these connections. AI researchers heavily favor Psychometrics & Judgment/Decision-Making (JDM) and Neural Mechanisms. In contrast, domains like Education and Social-Clinical psychology remain largely underexplored, despite their potential relevance to AI safety and alignment.

Patterns of Misengagement

The study identifies four primary ways in which AI research misapplies psychological concepts:

  1. Conceptual Overgeneralization: Complex, multi-faceted theories are often reduced to "catch-all" labels. For instance, in "Theory of Mind" (ToM) research, AI papers frequently conflate first-order tasks (knowing what someone else thinks) with second-order tasks (knowing what someone thinks someone else thinks), despite the vastly different cognitive demands involved.
  2. Partial Citation: Researchers tend to cite a few "classic" papers while ignoring decades of modern refinement and nuance within psychology.
  3. The "Telephone" Effect: A significant pitfall is the reliance on secondary citations. AI researchers often cite other AI papers' interpretations of psychology rather than the original source, creating a closed loop where misconceptions are reinforced and amplified.
  4. Misinterpretation: There is a frequent conflation of biological mechanisms (e.g., neural activity) with complex social behaviors, leading to category errors in how model outputs are explained.

Conclusion: Toward Collaborative Parity

To fix the "shaky bridge," the authors advocate for Collaborative Parity. Rather than "retrofitting" psychological theories to explain results after an experiment is finished, AI researchers should engage with psychologists from the project's inception.

Building a robust interdisciplinary infrastructure—including shared datasets and measurement maps—is essential. Only through genuine collaboration can we move beyond superficial analogies and develop a rigorous science of LLM behavior that respects the depth of both computer science and psychology.

Summary updated

Abstract

Social sciences have accumulated a rich body of theories and methodologies for investigating the human mind and behaviors, while offering valuable insights into the design and understanding of Artificial Intelligence (AI) systems. Focusing on psychology as a prominent case, this study explores the interdisciplinary synergy between AI and the field by analyzing 1,006 LLM-related papers published in premier AI venues between 2023 and 2025, along with the 2,544 psychology publications they cite. Through our analysis, we identify key patterns of interdisciplinary integration, locate the psychology domains most frequently referenced, and highlight areas that remain underexplored. We further examine how psychology theories/frameworks are operationalized and interpreted, identify common types of misapplication, and offer guidance for more effective incorporation. Our work provides a comprehensive map of interdisciplinary engagement between AI and psychology, thereby facilitating deeper collaboration and advancing AI systems.
BibTeX
@misc{jiang2025incomplete,
  author = {Han Jiang and Pengda Wang and Xiaoyuan Yi and Xing Xie and Ziang Xiao},
  title = {{The Incomplete Bridge: How AI Research (Mis)Engages with Psychology}},
  year = {2025},
  eprint = {2507.22847},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.22847},
}
Figure