ELM-Based Active Learning via Asymmetric Samplers: Constructing a Multi-Class Text Corpus for Emotion Classification

Xuefeng Shi, Min Hu, Fuji Ren, Piao Shi, Xiao Sun

TL;DR

This paper introduces a novel active learning (AL) approach using Extreme Learning Machines (ELMs) and "asymmetric samplers" to efficiently build a high-quality, multi-class Chinese text corpus for emotion classification. By intelligently selecting informative samples, their method significantly improves emotion classification performance while reducing the costly manual labeling effort.

What Problem Does This Paper Address?

Creating large, well-annotated datasets for deep learning models is a major bottleneck, especially for nuanced tasks like emotion classification. Labeling text data is time-consuming, expensive, and requires expert knowledge. Existing active learning methods, while helpful, often struggle with long training times or don't fully capture the complex nature of textual emotions. This paper aims to address the challenge of acquiring sufficient labeled data for Chinese text emotion classification by proposing an efficient AL strategy that leverages the strengths of ELMs and a novel sampling technique.

Methods

The core of the proposed method lies in two key components:

  • Extreme Learning Machines (ELMs): ELMs are used as the feature extractor. Their advantage is their rapid training time and ability to generate random hidden layer parameters, which bypasses the need for manual tuning and significantly speeds up the process compared to traditional deep learning models.
  • Asymmetric Samplers: This is the novel query strategy. It combines two crucial aspects for selecting the most informative unlabeled samples:
  • Uncertainty: Identifying samples that the model is least confident about classifying.
  • Representativeness: Selecting samples that best represent the overall distribution of the data. The asymmetric sampler uses a combination of cross-entropy, Kullback-Leibler divergence, and Earth Mover's distance to measure these criteria. A unique label balancing strategy is also introduced to ensure a more even distribution of emotion classes in the updated training set.

Datasets

  • Training and Testing Data: The experiments utilized sentence-level Chinese text data from the Weibo platform.
  • Emotion Taxonomy: The emotion classification was based on a peculiar emotion sorting technique developed by Ren Lab, which includes eight basic emotions (anxiety, anger, sorrow, hate, joy, love, expect, surprise) plus a neutral class.
  • Initial Labeled Data: The study started with an initial labeled dataset of 2022 sentence-level texts for training (Dtrain) and 1589 for testing (Dtest), with a nearly balanced distribution of emotion labels, except for the neutral class.

Key Insights and Ideas

  • ELM as an Efficient Feature Extractor for AL: The paper demonstrates that ELM, despite its simpler architecture, is effective for feature extraction in an active learning context, offering a speed advantage over more complex deep learning models.
  • The Power of Asymmetric Sampling: Combining uncertainty and representativeness in a single sampling strategy proved more effective than using either criterion alone. This suggests that a balanced approach is crucial for selecting truly informative samples.
  • Importance of Label Balancing: The proposed label balancing strategy within the asymmetric sampler is key to creating a more robust and representative corpus, preventing bias towards certain emotion classes.
  • Chinese Text Nuances: The paper highlights the implicit nature of Chinese emotional expression, which requires careful linguistic analysis and a robust model to capture effectively.

Key Results

  • Performance Improvements: The ELM-based AL with asymmetric samplers achieved significant improvements in emotion classification. Specifically, precision, recall, and F1 scores saw increases of 7.17%, 6.31%, and 6.71%, respectively, compared to baseline methods.
  • Classifier Performance: The proposed method demonstrated effectiveness across different classifiers, including ELM, SVM, and LR, with ELM showing particularly strong results when trained on the corpus built by the asymmetric samplers.
  • Efficiency: The ELM-based approach significantly reduces the time and labor required for data labeling, a critical factor in building large-scale datasets.
  • Qualitative Analysis: WordCloud analysis revealed insights into high-frequency emotion words and how Chinese culture influences emotional expression, particularly the implicit nature of certain emotions.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the asymmetric sampler shows superiority, further research is needed to explore more effective implementations of the query strategy. The performance of the model on other languages or domains is not explicitly discussed.
  • Future Work: The authors plan to investigate more lightweight neural networks, introduce an auto-labeling module to further reduce manual effort, and conduct more research on correlations among different emotion groups and the influence of cross-performances in AL sampling tasks. They also aim to better grasp the nuances of Chinese textual semantics and syntax.

Summary updated

Abstract

A high-quality annotated text corpus is vital when training a deep learning model. However, it is insurmountable to acquire absolute abundant label-balanced data because of the huge labor and time costs needed in the labeling stage. To alleviate this situation, a novel active learning (AL) method is proposed in this paper, which is designed to scratch samples to construct multi-class and multi-label Chinese emotional text corpora. This work shrewdly leverages the superiorities, i.e., less learning time and generating parameters randomly possessed by extreme learning machines (ELMs), to initially measure textual emotion features. In addition, we designed a novel combined query strategy called an asymmetric sampler (which simultaneously considers uncertainty and representativeness) to verify and extract ideal samples. Furthermore, this model progressively modulates state-of-the-art prescriptions through cross-entropy, Kullback–Leibler, and Earth Mover’s distance. Finally, through stepwise-assessing the experimental results, the updated corpora present more enriched label distributions and have a higher weight of correlative emotional information. Likewise, in emotion classification experiments by ELM, the precision, recall, and F1 scores obtained 7.17%, 6.31%, and 6.71% improvements, respectively. Extensive emotion classification experiments were conducted by two widely used classifiers—SVM and LR—and their results also prove our method’s effectiveness in scratch emotional texts through comparisons.
BibTeX
@inproceedings{shi2022elm,
  author = {Xuefeng Shi and Min Hu and Fuji Ren and Piao Shi and Xiao Sun},
  title = {{ELM-Based Active Learning via Asymmetric Samplers: Constructing a Multi-Class Text Corpus for Emotion Classification}},
  booktitle = {Symmetry},
  year = {2022},
  url = {https://openreview.net/forum?id=WqLYDxQ8lN},
}
Figure