Transforming Scholarly Landscapes: Influence of Large Language Models on Academic Fields beyond Computer Science

Aniket Pramanick, Yufang Hou, Saif M. Mohammad, Iryna GurevycharXiv 2024 · DOI · arXiv

Introduction

The rapid advancement of Large Language Models (LLMs) has sparked a transformation across the academic spectrum. While their origins are rooted in Natural Language Processing (NLP) and Computer Science (CS), their influence has increasingly "leaked" into diverse disciplines. This study provides a systematic empirical analysis of how LLMs are permeating 22 non-CS fields, identifying early adopters, usage patterns, and the critical "ethics gap" in current research.

Methodology

The researchers utilized a multi-stage pipeline to quantify inter-field influence:

  • Citation Mapping: Tracking citations from non-CS papers to 106 foundational LLM works (e.g., BERT, GPT-3, LLaMA).
  • Metric Analysis: Employing the Gini Index to measure adoption inequality and Age-of-Citation (AoC) to assess the "freshness" of models used.
  • Content Mining: Using NLP techniques to extract specific tasks and manual annotation to identify discussions of ethical risks (e.g., bias, hallucinations).
  • Data Source: Analysis of ~148k papers and ~273k citations from the Semantic Scholar (S2) database, spanning 2018 to early 2024.

Key Insights: The Drivers of Adoption

The study identifies Linguistics, Engineering, and Medicine as the "Big Three" adopters, collectively accounting for a significant portion of LLM-related research outside CS. Linguistics and Engineering alone represent approximately 45% of all non-CS LLM citations.

Figure 3: Sankey diagrams illustrating (a) the distribution of LLM citations across non-CS fields and (b) the percentage of non-CS collaborators, highlighting the role of CS authors as technological vectors.

A crucial finding is the role of interdisciplinary collaboration. There is a moderate correlation (0.38) between the presence of CS authors and the likelihood of LLM adoption in a paper. This suggests that computer scientists often act as "vectors," carrying technological expertise into other domains. Furthermore, non-CS researchers show a strong preference for "off-the-shelf" models like BERT, prioritizing stability and ease of use over the complex fine-tuning common in CS.

Breadth of Influence and the Ethics Gap

The adoption of LLMs is becoming increasingly widespread. The Gini Index, a measure of inequality, has shown a downward trend over the last several years, indicating that LLM influence is spreading more evenly across academia compared to previous technologies like RNNs.

Figure 6: The Gini Index of LLM citations across fields from 2018 to 2023, demonstrating a trend toward broader and more diverse adoption.

Despite this broad adoption, a significant "ethics blind spot" exists. Only about 2% of papers citing LLMs explicitly discuss inherent risks such as hallucinations or algorithmic bias. While the CS community is deeply engaged in safety debates, other fields often treat these models as "black box" tools, potentially overlooking critical limitations.

Conclusion

LLMs have moved beyond being a niche CS interest to become a foundational tool for global research. From protein structure prediction in Biology to judgment prediction in Law, the applications are vast. However, the lack of ethical scrutiny in non-CS fields remains a concern. Future efforts should focus on developing domain-specific LLMs and "Responsible AI" frameworks that empower non-technical researchers to use these powerful tools safely and effectively.

Summary updated

Abstract

Large Language Models (LLMs) have ushered in a transformative era in Natural Language Processing (NLP), reshaping research and extending NLP's influence to other fields of study. However, there is little to no work examining the degree to which LLMs influence other research fields. This work empirically and systematically examines the influence and use of LLMs in fields beyond NLP. We curate $106$ LLMs and analyze $\sim$$148k$ papers citing LLMs to quantify their influence and reveal trends in their usage patterns. Our analysis reveals not only the increasing prevalence of LLMs in non-CS fields but also the disparities in their usage, with some fields utilizing them more frequently than others since 2018, notably Linguistics and Engineering together accounting for $\sim$$45\%$ of LLM citations. Our findings further indicate that most of these fields predominantly employ task-agnostic LLMs, proficient in zero or few-shot learning without requiring further fine-tuning, to address their domain-specific problems. This study sheds light on the cross-disciplinary impact of NLP through LLMs, providing a better understanding of the opportunities and challenges.
BibTeX
@misc{pramanick2024transforming,
  author = {Aniket Pramanick and Yufang Hou and Saif M. Mohammad and Iryna Gurevych},
  title = {{Transforming Scholarly Landscapes: Influence of Large Language Models on Academic Fields beyond Computer Science}},
  year = {2024},
  eprint = {2409.19508},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.19508},
}
Figure