CHUCKLE -- When Humans Teach AI To Learn Emotions The Easy Way

Ankush Pratap Singh, Houwei Cao, Yong LiuarXiv 2025 · DOI · arXiv

Abstract

Curriculum learning (CL) structures training from simple to complex samples, facilitating progressive learning. However, existing CL approaches for emotion recognition often rely on heuristic, data-driven, or model-based definitions of sample difficulty, neglecting the difficulty for human perception, a critical factor in subjective tasks like emotion recognition. We propose CHUCKLE (Crowdsourced Human Understanding Curriculum for Knowledge Led Emotion Recognition), a perception-driven CL framework that leverages annotator agreement and alignment in crowd-sourced datasets to define sample difficulty, under the assumption that clips challenging for humans are similarly hard for machine learning models. Empirical results suggest that CHUCKLE increases the relative mean accuracy by 6.56% for LSTMs and 1.61% for Transformers over non-curriculum baselines, while reducing the number of gradient updates, thereby enhancing both training efficiency and model robustness.

Introduction

Speech Emotion Recognition (SER) is a notoriously noisy and subjective machine learning task. How a person speaks is influenced by cultural differences, context, and individual variability, making it difficult for models to generalize. To address this, researchers often employ Curriculum Learning (CL)—a training strategy inspired by human education where a model is fed "easy" examples before "hard" ones.

However, existing CL methods for SER typically define difficulty using signal-to-noise ratios or model-based mathematical metrics. These methods ignore the most natural metric for a subjective task: human perception. If humans cannot agree on whether a voice sounds "angry" or "sad," a model will likely struggle as well. This paper introduces the CHUCKLE framework to bridge this gap by using crowd-sourced human annotator agreement to explicitly define what makes an audio clip easy or hard to learn.

The CHUCKLE Framework

CHUCKLE (Crowdsourced Human Understanding Curriculum for Knowledge Led Emotion Recognition) shifts the focus from model-based heuristics to perception-driven difficulty.

The CHUCKLE framework pipeline: from crowdsourced human annotations to improved model results.

The framework utilizes a rule-based curriculum that categorizes data into four distinct bins based on the relationship between the actor's intended emotion and the crowd's perceived emotion:

  1. Clear Match (Easy): The majority of raters agree with each other and match the actor's intent.
  2. Clear Mismatch (Confidently Incorrect): The majority of raters agree with each other, but disagree with the actor's intent.
  3. Ambiguous Match: Raters are split, but some match the actor's intent.
  4. Ambiguous Mismatch (Tough): Raters are split, and none match the actor's intent.

Models are trained in stages, starting with the "Easy" bin and cumulatively adding harder bins to prevent catastrophic forgetting.

Experimental Methodology

The study utilizes the CREMA-D dataset, a standard audiovisual benchmark containing 7,442 clips from 91 actors. CREMA-D is uniquely suited for this work as it provides both the actor's intended label and multiple perceived labels from 2,443 crowd-sourced raters. To accommodate compute constraints, the authors trained LSTMs and Transformers on a 130-dimensional subset of the ComParE acoustic feature set rather than raw audio.

Results and Analysis

The empirical results demonstrate that human difficulty maps effectively to AI difficulty. Forcing a model to learn ambiguous emotional utterances early on introduces harmful noise; starting with universally agreed-upon emotions helps the model build a stable foundation of low-level acoustic features.

Training efficiency comparison: CHUCKLE strategies vs. non-curriculum baselines for LSTMs and Transformers.

Key Findings:

  • Accuracy and Efficiency: Using the rule-based curriculum, LSTMs saw a 6.56% relative increase in mean macro accuracy over non-curriculum baselines, while requiring 40.3% fewer gradient updates. Transformers achieved a 1.61% relative boost with 17.6% fewer updates.
  • Agreement vs. Alignment: The most successful curriculum prioritized human agreement over alignment with the ground truth. It was more effective to feed the model "Clear Mismatch" data (where humans strongly agreed on the wrong emotion) before "Ambiguous Match" data. A consistent, unambiguous signal is easier for early-stage models to process than pure ambiguity.
  • Model Sensitivity: While LSTMs adapted smoothly to new stages, Transformers showed temporary instability (spikes in training loss) when harder data was introduced, likely due to their sensitivity to data distribution shifts.

Conclusion

CHUCKLE demonstrates that perception-driven curriculum learning significantly enhances both the performance and training efficiency of SER models. By aligning machine learning with human perception, we can build more robust systems for subjective tasks. Future work will extend these curricula to other modalities, such as facial expressions, and validate findings on larger datasets using end-to-end deep learning.

Summary updated

Abstract

Curriculum learning (CL) structures training from simple to complex samples, facilitating progressive learning. However, existing CL approaches for emotion recognition often rely on heuristic, data-driven, or model-based definitions of sample difficulty, neglecting the difficulty for human perception, a critical factor in subjective tasks like emotion recognition. We propose CHUCKLE (Crowdsourced Human Understanding Curriculum for Knowledge Led Emotion Recognition), a perception-driven CL framework that leverages annotator agreement and alignment in crowd-sourced datasets to define sample difficulty, under the assumption that clips challenging for humans are similarly hard for machine learning models. Empirical results suggest that CHUCKLE increases the relative mean accuracy by 6.56% for LSTMs and 1.61% for Transformers over non-curriculum baselines, while reducing the number of gradient updates, thereby enhancing both training efficiency and model robustness.
BibTeX
@misc{singh2025chuckle,
  author = {Ankush Pratap Singh and Houwei Cao and Yong Liu},
  title = {{CHUCKLE -- When Humans Teach AI To Learn Emotions The Easy Way}},
  year = {2025},
  eprint = {2510.09382},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.09382},
  howpublished = {arXiv},
}
Figure