A New Multi-Domain Benchmark for Micro-Action Recognition and Detection

Yanbin Hao, Pengyu Liu, Xing Wei, Xun Yang, Dan Guo, Meng WangarXiv 2026 · arXiv

TL;DR

The paper introduces MMA-82, a large-scale, multi-domain benchmark designed to shift micro-action analysis from pristine laboratory environments into complex, real-world scenarios. Expanding the label space to 82 fine-grained categories across 79,574 annotated instances, the benchmark establishes rigorous baselines for micro-action recognition and detection while demonstrating that these subtle body movements provide vital, complementary cues for video-based emotion recognition.

What Problem Does This Paper Address?

While computer vision has made massive strides in recognizing macro-actions (e.g., running, jumping, or cooking), micro-actions remain largely underexplored. Micro-actions are short-duration, low-amplitude, subtle body movements—such as a fleeting shrug, a nervous leg twitch, or a quick nose touch—that unconsciously betray a person's internal emotional state or latent intentions.

Recognizing these movements is exceptionally difficult because they are brief (often lasting only a few frames), possess tiny spatial amplitudes, and are easily drowned out by background clutter, camera movement, or normal body posture adjustments. Furthermore, prior datasets like the authors' previous MA-52 benchmark were restricted to controlled, single-domain laboratory settings with fixed viewpoints and stable lighting. MMA-82 addresses this gap by introducing real-world environmental complexity, domain shifts, and multi-label co-occurrences.

MMA-82 Dataset Overview and Comparison

Methods

The authors constructed MMA-82 through a multi-stage pipeline:

  1. Taxonomy Expansion: They expanded the original 52 categories from MA-52 to 82 fine-grained categories, organized hierarchically into 7 coarse-grained body groups (e.g., Body-Hand, Leg-Hand, Head).
  2. Rigorous Annotation: To handle the highly subjective nature of temporal boundaries in subtle motions, each video was labeled by three trained annotators. Final ground-truth segments were determined using an Intersection-over-Union (IoU) merging strategy.
  3. Task Frameworks: They established two core tasks:
  4. Micro-Action Recognition (MAR): Classifying trimmed video clips under both In-Domain and Cross-Domain (Zero-Shot and Few-Shot) protocols.
  5. Multi-label Micro-Action Detection (MAD): Localizing and classifying multiple co-occurring micro-actions in untrimmed videos.
  6. Emotion-Action Mapping: They utilized a One-Versus-Neutral (OVN) decision tree strategy with TF-IDF features to isolate which micro-actions correlate most strongly with specific emotional states.

Datasets

The paper introduces the MMA-82 dataset, which aggregates 79,574 annotated instances from 454 subjects, totaling 75.87 hours of video. It spans four distinct domains:

  • Laboratory Interviews: Pristine, semi-controlled psychological interviews.
  • Psychiatric Patient Interviews: Publicly available clinical self-narratives on YouTube, capturing realistic behavioral expressions.
  • Street Interviews: Unscripted, highly unstructured real-world interviews featuring environmental noise, camera shake, and diverse lighting.
  • Emotion-rich TV Videos: Subsets derived from the CAER dataset, explicitly linking micro-actions to strong emotional expressions.

MMA-82 is split into two specialized evaluation sets: MMA-82-Rec (for recognition) and MMA-82-Det (for temporal detection).

MMA-82 Class Distribution and Statistics

Key Insights and Ideas

  • Context and Appearance Matter: RGB-based models consistently outperform skeleton-based models on micro-actions. This indicates that surface-level visual texture, clothing deformations, and precise hand-to-face contact context are crucial for detecting subtle movements—details that are completely lost in sparse skeleton joint coordinates.
  • The Cross-Domain Cliff: Models trained in controlled lab environments experience a massive performance drop when evaluated on street or clinical data. This highlights that current action recognition architectures lack robust generalization capabilities for subtle motions under domain shift.
  • Quantifying Body Language: The data validates classical psychological frameworks like Laban Movement Analysis (LMA). Positive emotions statistically trigger upward, open, and rhythmic movements (e.g., nodding, shrugging), while negative emotions manifest as contractive or downward movements (e.g., bowing the head).
  • Beyond Facial Expressions: Fusing body micro-actions with facial micro-expressions yields a substantial performance boost in emotion recognition, proving that body language provides vital, non-redundant affective signals when facial cues are ambiguous or obscured.
Sankey-style visualization mapping micro-actions to emotions

Key Results

  • In-Domain Recognition: On the full MMA-82-Rec benchmark, the RGB-based GC-TSM model achieved a Top-1 action-level accuracy of 60.43%, outperforming the skeleton-based PoseConv3D (56.62%). Performance was highest on lab data but dropped significantly on TV and street videos.
  • Cross-Domain Generalization: When a PoseConv3D model trained on lab data was evaluated in a zero-shot manner on psychiatric interviews, its Top-1 accuracy plummeted from an in-domain high of 64.84% down to 27.30%. Few-shot adaptation (up to 10-shot) only recovered performance up to 30.13%.
  • Temporal Detection: For the MAD task using the AdaTAD framework, scaling the backbone from VideoMAE-S to VideoMAE-L improved the average mAP from 18.77% to 23.41%. However, scaling further to VideoMAE-H caused a slight performance drop (23.07% mAP) due to optimization difficulties.
  • Emotion Recognition Boost: Fusing facial features (DeepFace) with micro-action features (TSM) boosted emotion classification accuracy to 32.86%, compared to just 22.86% when using facial micro-expressions alone.

Limitations and Future Work

The authors note that MMA-82 naturally suffers from a highly long-tailed category distribution (head and upper-limb movements heavily dominate everyday human behavior), making it difficult for models to learn rare action classes. Additionally, scaling up to massive video backbones (like VideoMAE-H) introduced optimization challenges without clear performance gains.

Future work will focus on developing specialized architectures that can better handle extreme domain shifts, mitigate class imbalance, and explore self-supervised pre-training strategies tailored specifically for low-amplitude, fleeting human motions.

Summary updated

Abstract

Micro-actions are short-duration, low-amplitude subtle body movements at the whole-body level that can reveal latent intentions, involuntary reactions, and fine-grained affective changes. Our previous MA-52 benchmark has provided an important foundation for micro-action recognition, but it remains limited in scale, scene diversity, task coverage, and evaluation protocols. To advance micro-action analysis toward more realistic and comprehensive settings, we introduce MMA-82, a large-scale multi-domain extension of MA-52. MMA-82 expands the label space from 52 to 82 fine-grained micro-action categories and covers four distinct domains, including laboratory interviews, street interviews, psychiatric patient interviews, and emotion-rich television videos, resulting in 77,856 annotated instances from 454 subjects. Built upon MMA-82, we establish two core tasks: Micro-Action Recognition and Multi-label Micro-Action Detection. For recognition, we further define in-domain and cross-domain protocols, including few-shot and zero-shot settings, to evaluate model robustness, transferability, and generalization. Extensive experiments show that current methods still struggle with realistic micro-action understanding, especially under domain shift, long-tailed category distributions, and complex temporal localization. Beyond benchmarking, we investigate the relationship between micro-actions and emotion, showing that micro-actions are strongly associated with emotional states and provide complementary cues to facial micro-expressions for improved emotion recognition. These results demonstrate that MMA-82 serves as a comprehensive and challenging benchmark for realistic micro-action analysis and a valuable resource for human-centered AI. MMA-82 is available at https://lpynow.github.io/MMA-82-AIM/.
BibTeX
@misc{hao2026new,
  author = {Yanbin Hao and Pengyu Liu and Xing Wei and Xun Yang and Dan Guo and Meng Wang},
  title = {{A New Multi-Domain Benchmark for Micro-Action Recognition and Detection}},
  year = {2026},
  eprint = {2606.14096},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.14096},
}

Loading related content...

Figure