Understanding influential factors for college instructors’ adoption of LLM-based applications using analytic hierarchy process

Xieling Chen, Lin Xu, Di Zou, Haoran Xie, Fu Lee WangJournal of Computers in Education 2025 · DOI

Abstract

With the accessibility of advanced artificial intelligence (AI)-based tools, particularly large language models (LLMs) such as ChatGPT, integrating LLMs into higher education has been considered a transformative shift in educational paradigms. However, instructors have numerous objections against the adoption of LLM-based applications. To promote the proper adoption of LLM-based applications for Chinese college instructors, this study investigates and assesses factors that affect instructors’ adoption. Specifically, this study proposes a multi-criteria decision-making model drawing upon technology acceptance theories such as the value-based adoption model to determine four key influential factors and their sub-factors. After collecting expert data from 22 Chinese college instructors with experience in integrating AI applications into classrooms across seven provinces, an analytic hierarchy process is adopted to weigh and prioritize these factors. Results show that “Usefulness” is the most important factor for encouraging instructors’ use of LLM-based applications, while “Effort” is of less concern. Among the sub-factors, “Effectiveness” and “Efficiency” are of intermediate importance in LLM-based application adoption, while “Perceived fee” has the least influence. Based on the findings, the study provides insights into Chinese college instructors’ adoption experiences of LLM applications as well as suggestions for promoting LLMs’ integration into instruction.

Introduction

While students have rapidly embraced Large Language Models (LLMs), faculty adoption has remained more cautious. Most existing research focuses on the learner's perspective, leaving a gap in understanding what motivates the individuals responsible for curriculum design. This study addresses the "instructor's dilemma"—the trade-off between the transformative potential of AI and the non-monetary costs, such as the time required for mastery and the technical complexity of prompt engineering. By moving beyond anecdotal evidence, this research provides a weighted, hierarchical model of instructor motivation.

Methodology

The researchers employed the Analytic Hierarchy Process (AHP), a multi-criteria decision-making approach, to quantify the factors driving adoption. The process followed a structured sequence from hierarchy establishment to priority factor calculation.

Flowchart of the Analytic Hierarchy Process (AHP) procedure used in the study.

The methodology involved three primary phases:

  1. Theoretical Framework: Synthesis of the Technology Acceptance Model (TAM), the Value-based Adoption Model (VAM), and the Unified Theory of Acceptance and Use of Technology (UTAUT).
  2. Hierarchical Modeling: Organization of 15 sub-factors into four primary "Criterion Layers": Usefulness, Enjoyment, Technicality, and Effort.
  3. Expert Pairwise Comparison: Data collection from 22 expert instructors who performed pairwise comparisons to determine the relative weights of each factor.

Data and Participants

The study utilized opinion data from 22 Chinese college instructors across seven provinces. The participant profile included:

  • Demographics: 90% of participants were over 30 years old.
  • Experience: 86% reported positive effects from LLM use, with experience in tools such as ChatGPT, Baidu's ERNIE Bot, and IFlytek Spark.
  • Validation: A "Consistency Ratio" (CR) was applied to ensure the logical consistency of the experts' pairwise judgments.

Analysis and Insights

The analysis revealed several critical drivers for adoption:

  • Utility is Paramount: "Usefulness" (52.2%) carries more weight than the other three categories combined. Instructors prioritize tools that demonstrably improve teaching quality or reduce workload.
  • The Psychological Driver of Pleasure: Within the "Enjoyment" category, Pleasure (immediate user experience) was nearly twice as important as Satisfaction (long-term goals), suggesting that smooth, rewarding interactions are key to continued use.
  • Complexity over Flexibility: Instructors are more concerned with the difficulty of mastering a tool (Complexity) than its cross-device availability (Flexibility).
  • Low Sensitivity to Cost: "Perceived Fee" was the lowest-ranked sub-factor (2.5% global weight). Technically literate instructors appear willing to pay for premium services if the perceived effectiveness is high.

Results

The AHP analysis produced global weights that rank the importance of each factor in the total decision-making process:

Rank Factor Global Weight Category
1 Effectiveness 24.47% Usefulness
2 Efficiency 14.43% Usefulness
3 Pleasure 13.40% Enjoyment
4 Complexity 10.65% Technicality
5 Perceived Time 7.85% Effort
... ... ... ...
10 Perceived Fee 2.53% Effort

The top two factors—Effectiveness and Efficiency—account for nearly 40% of the total weight, confirming that instructors view LLMs primarily as productivity and quality-enhancement tools.

Conclusion

While providing valuable insights, the study notes limitations regarding its sample size (22 experts in China) and its focus on conscious perceptions. Future research should consider longitudinal tracking to observe how these weights shift as AI becomes more integrated into educational environments. As LLM interfaces become more intuitive, the barriers related to "Technicality" may decrease, further shifting the focus toward the utility and effectiveness of these tools in the classroom.

Summary updated

Abstract

Abstract With the accessibility of advanced artificial intelligence (AI)-based tools, particularly large language models (LLMs) such as ChatGPT, integrating LLMs into higher education has been considered a transformative shift in educational paradigms. However, instructors have numerous objections against the adoption of LLM-based applications. To promote the proper adoption of LLM-based applications for Chinese college instructors, this study investigates and assesses factors that affect instructors’ adoption. Specifically, this study proposes a multi-criteria decision-making model drawing upon technology acceptance theories such as the value-based adoption model to determine four key influential factors and their sub-factors. After collecting expert data from 22 Chinese college instructors with experience in integrating AI applications into classrooms across seven provinces, an analytic hierarchy process is adopted to weigh and prioritize these factors. Results show that “Usefulness” is the most important factor for encouraging instructors’ use of LLM-based applications, while “Effort” is of less concern. Among the sub-factors, “Effectiveness” and “Efficiency” are of intermediate importance in LLM-based application adoption, while “Perceived fee” has the least influence. Based on the findings, the study provides insights into Chinese college instructors’ adoption experiences of LLM applications as well as suggestions for promoting LLMs’ integration into instruction.
BibTeX
@article{chen2025understanding,
  author = {Xieling Chen and Lin Xu and Di Zou and Haoran Xie and Fu Lee Wang},
  title = {{Understanding influential factors for college instructors’ adoption of LLM-based applications using analytic hierarchy process}},
  journal = {Journal of Computers in Education},
  year = {2025},
  doi = {10.1007/s40692-025-00363-0},
}
Figure