Navigating Cognitive Engagement in AI-Enhanced Education: Lexical Diversity and Open-Ended Inquiry in Journalism Learning

Jeanne McClure, Franziska Bickel, Shiyan Jiang, Jie Chao, Carolyn Penstein RoséProceedings 2024 · DOI

Abstract

This study investigates the relationship between students' writing patterns—specifically lexical diversity (LD)—and their cognitive engagement (CE) within an AI-enhanced journalism curriculum. By analyzing 840 open-ended responses from high school students, the researchers found that while high-level thinking often correlates with a richer vocabulary, total word count (TNW) is a significantly stronger predictor of deep cognitive engagement than vocabulary variety alone. The findings suggest that linguistic metrics like LD should be used cautiously as proxies for critical thinking.

Introduction

As Artificial Intelligence (AI) reshapes professional landscapes, educators are increasingly integrating AI literacy into high school curricula. A central challenge in this transition is fostering Cognitive Engagement (CE)—the psychological investment and effort students dedicate to their learning. To stimulate CE, teachers frequently employ open-ended questions. Researchers, in turn, often use Lexical Diversity (LD)—a measure of vocabulary richness—as a quantitative proxy for the depth of student engagement. This study seeks to validate whether such linguistic metrics truly reflect a student's cognitive processes in technology-rich environments.

Methodology

The researchers employed an explanatory sequential mixed-methods design, analyzing responses from 28 high school students (primarily 11th and 12th graders) in a US journalism course.

Tools and Frameworks

Students utilized StoryQ, an educational platform for exploring machine learning (ML) concepts like sentiment analysis without coding. Their responses were mapped to two primary educational frameworks:

  1. ICAP Model: Categorizing engagement into Passive, Active, Constructive, and Interactive modes.
  2. Revised Bloom’s Taxonomy: Ranking cognitive tasks from "Remembering" to "Creating."

Data Analysis

The study analyzed 840 written responses across three ML modules. Quantitative methods included K-means clustering to group students based on completion rates, CE scores, and the Measure of Textual Lexical Diversity (MTLD). Additionally, a multinomial logistic regression was performed to identify the best predictors of engagement levels. Qualitative deep-dives into specific responses provided context for the quantitative findings.

Results and Discussion

The study revealed a complex relationship between how students write and how they think.

The "Vocabulary Trap" and the Power of Quantity

A key insight was that high lexical diversity does not automatically equate to high-level thinking. Qualitative analysis showed that students might use a wide variety of words simply to list items (Passive engagement) without synthesizing information. Surprisingly, the Total Number of Words (TNW) was a more significant predictor of engagement than vocabulary variety. Each additional word written increased the odds of a student reaching "Active" or "Constructive" engagement levels by 20% and 30%, respectively.

Statistical Significance of Lexical Diversity

Quantitative analysis confirmed that lexical diversity varies significantly with engagement levels. A Welch ANOVA showed significant differences in MTLD scores across different cognitive engagement categories ($p < .001$).

Mean Lexical Diversity (LD) values across different levels of Cognitive Engagement.

However, when controlled for other factors in the regression model, LD's predictive power was not statistically significant at the .05 level. This indicates that while LD is a useful descriptive tool, it should not be used in isolation to assess the depth of student thought.

Student Profiles

Clustering identified three distinct student archetypes:

  • Cluster 0: Characterized by low lexical diversity and predominantly passive engagement.
  • Cluster 1: Showed moderate diversity and engagement with varied completion rates.
  • Cluster 2: Displayed high lexical diversity and high active/constructive engagement, showing a strong ability to infer new knowledge.

Conclusion

While linguistic metrics like Lexical Diversity offer insights into student writing, they are not definitive indicators of cognitive depth. The study highlights that the sheer volume of output (TNW) may be a more reliable indicator of a student's psychological investment in open-ended tasks.

Limitations and Future Directions

The study's small sample size (n=28) and the sensitivity of LD metrics to short texts are noted limitations. Future research should expand to larger, more diverse populations and different subject areas. Pedagogically, the focus should shift from merely encouraging "more words" to strategies that prompt students to move from information listing to the synthesis and transfer of knowledge across new contexts.

Summary updated

BibTeX
@inproceedings{mcclure2024navigating,
  author = {Jeanne McClure and Franziska Bickel and Shiyan Jiang and Jie Chao and Carolyn Penstein Rosé},
  title = {{Navigating Cognitive Engagement in AI-Enhanced Education: Lexical Diversity and Open-Ended Inquiry in Journalism Learning}},
  booktitle = {Proceedings},
  year = {2024},
  doi = {10.22318/icls2024.875205},
}
Figure