CALM-AH: An ABAW11-Calibrated Multimodal Ensemble with Reliability-Gated Multi-Expert Consensus for Video-Level Ambivalence and Hesitancy Recognition

Wenzhuo Sun, Mingjian Liang, Richard Attfield, Zongyuan Ge, Xuelian Cheng, Pamela Carreno-MedranoarXiv 2026 · arXiv

TL;DR

Researchers have developed CALM-AH, a multimodal ensemble designed to detect subtle states of ambivalence and hesitancy in interview videos. By combining a diverse feature set with a novel Reliability-Gated Multi-Expert Consensus (RG-MEC)—which only overrides a stable "anchor" prediction when three distinct AI experts unanimously agree—the system achieved a top-tier Macro-F1 score of 0.7771 in the ABAW11 challenge.

What Problem Does This Paper Address?

Recognizing human emotions like "happiness" or "anger" is a well-trodden path in computer vision, but Ambivalence and Hesitancy (A/H) are far more elusive. These states are characterized by conflicting evaluations or delayed commitments, often expressed through subtle, non-verbal cues like a specific speaking rhythm, a micro-expression of discomfort, or linguistic hedging (e.g., "Um... yeah... maybe").

The challenge lies in the fact that these cues are often inconsistent across modalities. A speaker might sound confident but look uneasy, or vice versa. Furthermore, datasets for these specific behaviors are small, making high-capacity models prone to overfitting. The authors aim to build a system that robustly integrates text, audio, and video while resisting the "noise" of individual model errors.

Methods

The authors propose a two-tiered architecture:

  1. CALM-AH (The Base Ensemble): This system extracts four types of features:

  2. Textual: Encoded via F2LLM to catch semantic cues like self-correction.

  3. Acoustic: Encoded via HuBERT to capture prosody and pauses.
  4. Visual: Encoded via SigLIP2 to track facial activity.
  5. Behavioral Stats: 102 descriptors tracking visual consistency and silence patterns.

    They trained 45 candidate classifiers across 15 different modality combinations. Instead of simple averaging, they used weighted hard voting. Each model’s decision threshold was independently calibrated for Macro-F1, and their binary "votes" were combined to limit the influence of any single overconfident or poorly calibrated model.

  6. RG-MEC (The Consensus Logic): To further refine predictions, they introduced a "unanimity-gated" decision layer. As shown in the pipeline below, the system designates an initial prediction as a Stability Anchor. Three "Correction Experts"—CALM-AH, AffectGPT (multimodal reasoning), and a GPT-based semantic verifier—act as a committee. The anchor is only overridden if all three experts unanimously agree on a different label.

Fig. 1: The pipeline of our method.

Datasets

The study utilizes the BAH dataset provided for the 11th Affective & Behavior Analysis in-the-Wild (ABAW11) challenge. The dataset is partitioned by participant, ensuring that the model is tested on individuals it has never seen during training—a critical requirement for real-world generalization.

Table 1: BAH split statistics used in the ABAW11 video-level A/H task.

Key Insights and Ideas

  • The Power of Veto: The core "aha moment" is the move away from majority voting. In subtle behavior recognition, a 2-vs-1 split among experts is often just noise. By requiring unanimity to override the anchor, the system significantly reduces false positives and isolated expert errors.
  • Hard Voting vs. Soft Probabilities: The authors found that because different classifier families (MLPs, Random Forests, GBDTs) produce incomparable probability scales, converting their outputs to binary decisions (hard voting) after careful threshold calibration provides more reliable generalization.
  • Modality Synergy: The results confirm that while linguistic information is a strong signal for hesitancy, the integration of acoustic and visual "behavioral statistics" is essential for capturing the full context of ambivalence.

Key Results

The system was evaluated using Macro-F1 (MF1), which treats the recognition of A/H and No-A/H samples with equal importance.

  • Initial Anchor: 0.7485 MF1
  • Standalone CALM-AH: 0.7525 MF1
  • Full RG-MEC System: 0.7771 MF1

The jump to 0.7771 demonstrates that the "unanimity-gated" approach successfully filtered out errors that the individual models would have made in isolation. Notably, the system maintained high performance across both classes (A/H and No-A/H), proving it wasn't just biased toward the majority label.

Limitations and Future Work

The authors acknowledge that the RG-MEC logic was a "post-hoc, challenge-specific decision heuristic." While it performed exceptionally well, its reliance on a specific "Stability Anchor" means the system's baseline is only as good as its primary model.

Future work could involve:

  • Dynamic Weighting: Instead of fixed hard-voting weights, exploring weights that change based on the quality of the input (e.g., discounting visual votes if the face is obscured).
  • End-to-End Integration: Moving the consensus logic from the decision level into the training phase, though this risks the overfitting issues the authors were trying to avoid.

Summary updated

Abstract

Ambivalence and hesitancy (A/H) are subtle behavioural states that may be expressed through language, voice, facial activity, and other non-verbal cues. The ABAW11 A/H Video Recognition Challenge asks systems to assign a binary A/H label to each naturalistic interview video. Performance is measured using Macro-F1 so that recognition of both A/H and No-A/H samples receives equal importance. We present CALM-AH, a multimodal ensemble that combines textual, acoustic, visual, and derived behavioural-statistical features. We construct 15 non-empty combinations of these feature branches. For each combination, we select the best of three classifier families using validation binary cross-entropy and optimise its decision threshold for validation Macro-F1. The resulting binary decisions are combined using fixed hard-voting weights transferred from BROTHER. We further introduce Reliability-Gated Multi-Expert Consensus(RG-MEC), an anchor-preserving decision-level ensemble that combines an initial prediction with three complementary correction experts: CALM-AH, AffectGPT, and a GPT-based semantic verifier. The initial system provides the default prediction. Its label is overridden only when all three correction experts unanimously support the same alternative class; otherwise, the anchor prediction is retained. This unanimity-gated design limits the influence of isolated expert errors while permitting bidirectional correction when task-specific, multimodal-affective, and semantic-pragmatic evidence are fully consistent. On the participant-disjoint ABAW11 dataset, CALM-AH achieves a Macro-F1 of 0.7525, and the complete RG-MEC system achieves 0.7771.
BibTeX
@misc{sun2026calm,
  author = {Wenzhuo Sun and Mingjian Liang and Richard Attfield and Zongyuan Ge and Xuelian Cheng and Pamela Carreno-Medrano},
  title = {{CALM-AH: An ABAW11-Calibrated Multimodal Ensemble with Reliability-Gated Multi-Expert Consensus for Video-Level Ambivalence and Hesitancy Recognition}},
  year = {2026},
  eprint = {2607.29310},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.29310},
}
Figure