Teaching Interactively to Learn Emotions in Natural Language

Rajesh Titung, Cecilia AlmNAACL 2022

TL;DR

This paper explores "Machine Teaching" (MT), an interactive machine learning approach, for text-based emotion prediction. By comparing MT to Active Learning (AL) and traditional supervised methods, they demonstrate MT's potential for efficient learning, especially when fine-tuning pre-trained models, offering a promising direction for human-AI collaboration in NLP.

What Problem Does This Paper Address?

Accurately predicting emotions in text is a challenging task due to the inherent subjectivity of human interpretation. Factors like individual biases, cultural differences, and even fatigue can influence how emotions are labeled. This subjectivity, coupled with the scarcity of labeled data for many affective states, makes traditional supervised learning methods less efficient. Interactive Machine Learning (iML) offers a solution by allowing humans to directly guide the learning process, making it more resource-efficient. This paper specifically investigates "Machine Teaching" (MT), a less-studied iML paradigm where a human "teacher" actively selects data to train a model, contrasting it with the more established "Active Learning" (AL) where the model queries for labels. The goal is to assess the efficiency and effectiveness of MT for text-based emotion prediction, particularly in comparison to AL and offline methods.

Methods

The study compares three main approaches:

  • Offline Supervised Learning (All-in-one batch): This is the baseline, where the model is trained on the entire labeled dataset at once.
  • Active Learning (AL): The learner (model) iteratively selects instances it is most uncertain about to be labeled by a human "agent-user." The paper explores uncertainty sampling strategies like least confidence, margin sampling, and entropy.
  • Machine Teaching (MT): A human "agent-teacher" actively selects instances to label and train the learner. The paper simulates this using margin sampling-based AL as a teacher and explores three teaching criteria:
  • Error-based: The teacher provides the correct label for incorrect predictions.
  • Error-based with counting: Similar to error-based, but the teacher continues providing labels even for correct predictions for a period to ensure periodic model updates.
  • State change-based: The teacher provides a label if the current instance's real class differs from the previous instance's class label.

The experiments were conducted using a Linear SVM with hinge loss for initial comparisons and then fine-tuning a pre-trained BERT model for more advanced scenarios.

Datasets

The study uses a subset of the GoEmotions dataset (Demszky et al., 2020), which contains emotion labels for Reddit comments. The dataset exhibits class imbalance, as visualized in Figure 2. The training set comprised approximately 2800 instances, and the test set had about 700 instances.

Key Insights and Ideas

  • MT's Efficiency Potential: Machine Teaching, particularly with the state change criterion, shows significant promise in achieving high performance with substantially fewer labeled instances compared to traditional batch learning.
  • AL as a Strong Baseline: Active Learning, especially with uncertainty sampling strategies, also demonstrates considerable resource efficiency, often matching full batch performance with about half the data.
  • Fine-tuning Boosts Performance: Fine-tuning a pre-trained BERT model using iML (both AL and MT) leads to very high performance (close to 96%) and requires even less data for convergence than the SVM-based experiments.
  • State Change for Imbalanced Data: The state change-based MT criterion appears particularly effective in handling imbalanced datasets, outperforming other MT criteria in this study.
  • Human-in-the-Loop is Key: The research underscores the value of human expertise in guiding machine learning, especially for subjective tasks like emotion prediction, leading to more robust and interpretable models.

Key Results

  • AL vs. Batch: AL strategies achieved performance equivalent to full batch training using approximately half the data.
  • MT vs. Batch: MT, especially with the state change criterion, also demonstrated significant resource efficiency. The state change criterion outperformed error-based methods.
  • Fine-tuning BERT: When fine-tuning BERT, AL matched offline training performance using less than half the instances. The convergence for fine-tuning was also faster, requiring less data than the initial SVM experiments.
  • Computational Cost: MT with the state change criterion took the longest time (around 40 minutes) in the SVM experiments, highlighting a trade-off between teaching strategy complexity and computational resources.

Limitations and Future Work

The authors acknowledge several limitations:

  • Linguistic Characteristics: The study did not deeply examine the linguistic characteristics of the pre-trained models used.
  • Hybrid MT-AL: They did not deeply explore hybrid MT-AL strategies, though an AL approach was used as a teacher in MT.
  • Artificial Teacher: The MT teacher was simulated, not a real-time human agent.

Future work directions include:

  • Real-time Interactions: Conducting real-time user/teacher interactions.
  • Broader Teaching Criteria: Exploring a wider range of teaching criteria for MT.
  • New Instance Selection Forms: Investigating novel methods for training instance selection.
  • Understudied Affective States: Focusing on affective states that are less covered in existing corpora.
  • Specialized Teachers: Researching specialized teachers in NLP tasks for better data selection.
  • Multimodal Data: Extending findings to multimodal data.
  • Incremental Learning: Exploring incremental (online) learning versus non-incremental setups, noting the potential for catastrophic forgetting in incremental approaches but their efficiency for low-memory devices.

Summary updated

BibTeX
@inproceedings{titung2022teaching,
  author = {Rajesh Titung and Cecilia Alm},
  title = {{Teaching Interactively to Learn Emotions in Natural Language}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure