Demystify Adult Learning: A Social Network and Large Language Model Assisted Approach

Fang Liu, Bosheng Ding, Chong Guan, Zhang Wei, Dusit Niyato, Justina TanOpenReview 2024 · OpenReview

Introduction

Adult learners represent a unique and growing demographic in the global educational landscape. Unlike traditional students, these individuals must navigate a complex web of responsibilities, including full-time employment, family obligations, and financial constraints. Understanding their specific needs and sentiments is crucial for educational institutions to improve retention and learning outcomes. However, traditional feedback mechanisms, such as surveys, often suffer from low response rates and "polite" bias, where respondents provide socially acceptable rather than honest answers.

Social media platforms like Reddit, Twitter, and Tumblr have emerged as valuable repositories of candid, real-time feedback. Despite this potential, analyzing this data at scale presents significant challenges. General-purpose Large Language Models (LLMs) often struggle with the domain-specific jargon, slang, and nuanced emotional contexts prevalent in adult education discussions. To bridge this gap, this research introduces A-Learn, a specialized framework designed to fine-tune LLMs for high-accuracy sentiment analysis within the adult learning domain.

Methodology

The A-Learn framework utilizes a multi-stage pipeline designed to transform raw social media chatter into actionable educational insights.

Data Collection and Pre-processing

The researchers aggregated a dataset of over 55,000 comments from Reddit (31,724), Twitter (22,270), and Tumblr (1,043) spanning from 2011 to 2023. Data was filtered using keywords such as "adult learning," "continuous learning," and "work-study balance" to ensure relevance.

Hybrid Labeling Strategy

To address the scarcity of labeled data in this niche, the study employed a "data-lite" hybrid approach:

  1. LLM Labeling: GPT-3.5 was utilized to label an initial set of 1,000 comments as Positive, Neutral, or Negative.
  2. Human Verification: To establish a "gold standard" for evaluation, human experts manually verified a separate 300-comment test set, ensuring the model's performance was measured against high-quality, consensus-driven data.

Domain Adaptation and Fine-Tuning

The core of the framework involves fine-tuning several transformer-based architectures, including BERT, RoBERTa, and DistilBERT. The researchers conducted a sensitivity analysis to determine the optimal number of trainable layers. By unfreezing specific layers of the transformer encoder, the models were able to adapt to the unique linguistic patterns of adult learners while retaining the general language knowledge acquired during pre-training.

Key Results and Performance

The A-Learn framework demonstrated a significant performance leap over baseline models. The model based on Twitter-RoBERTa achieved a peak accuracy of 91.3%, representing an improvement of 15% to 24.3% compared to non-customized versions of the same architectures.

One of the most notable findings was the efficiency of the approach. Despite the high accuracy, the fine-tuning process required only approximately 10 minutes per model on an NVIDIA V100 GPU. This suggests that specialized educational sentiment analysis can be achieved without the prohibitive costs or time requirements of training models from scratch.

Discussion: Insights into the Adult Learner Experience

The Pitfalls of Synthetic Data

A critical insight from the study was the comparison between training on real social media comments versus synthetic data generated by LLMs. The researchers found that training on synthetic data actually decreased performance. While LLMs can generate grammatically correct text, they often fail to replicate the unstructured, slang-heavy, and "messy" nature of authentic social media discourse. This underscores the importance of using real-world data for domain-specific tasks.

Qualitative Sentiment Drivers

Word cloud analysis of the labeled data revealed distinct themes driving sentiment:

  • Negative Sentiments: Primarily driven by domestic and logistical struggles, with frequent mentions of "family vacation," "home alone," and "time job."
  • Positive Sentiments: Focused on professional advancement and specific learning resources, such as "work smarter" and "Khan Academy."

Model Sensitivity

The sensitivity analysis indicated that unfreezing all 12 layers of the transformer yielded the best results for domain adaptation. Freezing too many layers restricted the model's ability to learn the specific nuances of the adult learning vocabulary, which often differs significantly from general-purpose datasets.

Conclusion and Future Work

The A-Learn framework provides a robust, efficient, and highly accurate method for monitoring the sentiments of adult learners. By leveraging social media data and LLM-assisted labeling, researchers can gain a more honest understanding of the hurdles this demographic faces.

Future research will focus on expanding the "gold standard" human-verified dataset and incorporating data from professional platforms like LinkedIn. Ultimately, the goal is to move beyond identification and use these insights to drive educational interventions and policy changes that directly support the unique needs of adult students.

Summary updated

Abstract

Adult learning is increasingly recognized as a crucial way for personal development and societal progress. It however is challenging, and adult learners face unique challenges such as balancing education with other life responsibilities. Collecting feedback from adult learners is effective in understanding their concerns and improving learning experiences, and social networks provide a rich source of real-time sentiment data from adult learners. Machine learning technologies especially large language models (LLMs) perform well in automating sentiment analysis. However, none of such models is specialized for adult learning with accurate sentiment understanding. In this paper, we present A-Learn, which enhances adult learning sentiment analysis by customizing existing general-purpose LLMs with domain-specific datasets for adult learning. We collect adult learners' comments from social networks and label the sentiment of each comment with an existing LLM to form labelled datasets tailored for adult learning. The datasets are used to customize A-Learn from several base LLMs. We conducted experimental studies and the results reveal A-Learn's competitive sentiment analysis performance, achieving up to 91.3% accuracy with 20% improvement over the base LLM. A-Learn is also employed for word cloud analysis to identify key concerns of adult learners. The research outcome of this study highlights the importance of applying machine learning with educational expertise for teaching improvement and educational innovations that benefit adult learning and adult learners.
BibTeX
@misc{liu2024demystify,
  author = {Fang Liu and Bosheng Ding and Chong Guan and Zhang Wei and Dusit Niyato and Justina Tan},
  title = {{Demystify Adult Learning: A Social Network and Large Language Model Assisted Approach}},
  year = {2024},
  eprint = {2404.13267},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.13267},
}
Figure