QA-CMIF:Accurate and Lightweight Cross-Modal Latent Imputation and Fusion for Fault-Tolerant sEMG-Inertial Hand Gesture Recognition

Xinyu Tao, Shuang Huang, Yeyun Cai, Fang DengarXiv 2026 · arXiv

TL;DR

QA-CMIF is a lightweight, fault-tolerant framework for hand gesture recognition that can "repair" corrupted muscle sensor data in real-time. By processing sensor channels independently and using cross-attention to fill in missing information from other sensors, it maintains high accuracy even when half the sensors fail, all while running fast enough for seamless use in prosthetic controllers.

What Problem Does This Paper Address?

In a controlled laboratory setting, hand gesture recognition using surface electromyography (sEMG)—sensors that pick up electrical activity from muscles—works remarkably well. However, the "real world" is messy. For someone wearing a prosthetic limb or a gesture-controlled armband, sensors frequently fail. Electrodes lift off the skin, sweat causes signal drift, and wires fatigue and break.

Current AI models have a "one bad apple spoils the bunch" problem. Most multimodal fusion networks mix all sensor channels in their very first layer. If just one or two electrodes produce a high-amplitude "pop" or "shock" from losing skin contact, that noise contaminates the entire internal representation of the data, leading to catastrophic misclassifications. Existing fixes either simply turn off the bad channels (which loses vital information needed for complex gestures) or try to reconstruct the raw electrical waveforms (which is computationally expensive and slow).

Methods

The authors propose QA-CMIF (Quality-Aware Cross-Modal Latent Imputation and Fusion), which handles sensor failure through a four-stage pipeline:

  1. Channel-Independent Encoding: Unlike standard models, QA-CMIF processes each sEMG channel separately using "grouped convolutions." This ensures that noise on one electrode cannot leak into the data of another at the start.
  2. Multi-Statistic Soft Gating: The model calculates four lightweight statistics for each channel (like signal energy and "zero-count" proportion) to score its reliability. This "soft gate" identifies which channels are clean and which are garbage.
  3. Joint-Context Latent Imputation: This is the core innovation. Instead of trying to fix the raw, wiggly electrical signal, the model fixes the meaning of the signal in "latent space." It uses a cross-attention mechanism—similar to the tech inside Transformers—to look at the reliable sEMG channels and the data from the Inertial Measurement Unit (IMU/accelerometer) to "guess" what the missing muscle data should have been.
  4. Confidence-Weighted Residual Bypass: The model uses a clever bypass to ensure that high-quality, clean signals are passed through untouched, while only the "bad" tokens are replaced by the imputed (guessed) versions.

Datasets

The framework was validated on three major public benchmarks involving 29 participants:

  • Ninapro DB3: Data from 11 transradial amputees recorded at a high 2000 Hz sampling rate.
  • Ninapro DB2: Data from intact subjects using medical-grade arrays (2000 Hz).
  • Ninapro DB5: Data from intact subjects using consumer-grade Myo armbands at a lower 200 Hz sampling rate.

The tasks ranged from simple 17-class wrist movements to highly complex 52-class finger and grasp taxonomies.

Key Insights and Ideas

  • Isolation Before Interaction: By preventing sensors from "talking" to each other until their quality is verified, the model becomes inherently resilient to localized bursts of noise.
  • Muscle Synergy as a Map: The model succeeds because muscles don't work in isolation; they work in "synergies." If the model knows what five muscles are doing and knows the orientation of the arm (via the IMU), it has enough context to accurately infer what a sixth, "broken" sensor should be reporting.
  • The Complexity Bonus: A fascinating finding was that the "imputation" (the guessing of missing data) becomes more valuable as the task gets harder. For simple gestures, you can get away with missing data. For fine-grained finger movements, the "guessed" data provides the crucial edge needed to tell gestures apart.
  • Efficiency by Design: By performing imputation in the compact latent space rather than the raw signal space, the authors achieved a model that is 46%–55% smaller than current state-of-the-art baselines.

Key Results

  • Robustness: Under a grueling "50% channel corruption" test (where half the sensors are failing), QA-CMIF outperformed standard gating methods by up to 4.15 percentage points on complex tasks.
  • Superiority over Conventional Fusion: It surpassed standard multimodal fusion models by a massive 4.61 to 19.85 percentage points when sensors were noisy.
  • Lightweight Performance: The model uses only about 175k parameters.
  • Ultra-Low Latency: It boasts a 99th-percentile CPU latency of just 1.44 ms. For context, prosthetic control usually requires a response within 50 ms to feel "real-time" to the user, meaning QA-CMIF uses less than 3% of the available time budget.

Limitations and Future Work

While QA-CMIF is a major step forward, the authors note a few boundaries:

  • Context Dependency: The "repair" mechanism relies on having some good data. If nearly all sensors fail simultaneously (e.g., the armband is removed), the model's performance naturally degrades toward the baseline.
  • Synthetic vs. Real Noise: The model was trained using synthetic noise operators (simulating lift-off and bursts). Future work will need to test how it handles the slow, gradual "drift" of signals that happens over hours of wear.
  • Closed-Loop Testing: The current study is an offline analysis. The next logical step is deploying this on embedded hardware for real-time, "in-the-loop" control with amputee users to see how it feels in practice.

Summary updated

Abstract

Fusing surface electromyography (sEMG) with inertial measurements is widely used for myoelectric prosthesis control and wearable human-machine interfaces. In daily wear, however, socket micro-displacement, electrode lift-off, and wire failures often corrupt an unknown subset of sEMG channels through burst artifacts, lift-off shocks, or open-circuit dropouts. Existing fusion networks mix all channels in the first convolutional layer, so a few corrupted electrodes contaminate the entire latent representation, whereas gate-only attenuation of corrupted channels discards the co-activation cues needed to separate fine-grained gestures. We propose QA-CMIF, a quality-aware cross-modal latent imputation and fusion framework that encodes each sEMG channel independently, scores its reliability with a soft gate, and protects intact channels via a confidence-weighted residual bypass, while a cross-attention imputer repairs degraded channels in latent space using the inertial stream and remaining reliable sEMG channels under a bad-channel alignment loss. Across within-subject evaluations on 29 participants from Ninapro DB3, DB2, and DB5 spanning 17-52 gesture classes, QA-CMIF outperformed the gate-only baseline under 50% channel corruption by 1.29-2.65 percentage points, denoted pp, across all three 17-class cohorts with paired-t Holm-adjusted p<0.002, widened this margin to 2.59-4.15 pp on the 40- and 52-class tasks, and surpassed corruption-augmented conventional fusion by 4.61-19.85 pp. Requiring only 0.175-0.185M parameters and 2.45-2.63M multiply-accumulate operations across the 2000 Hz and 200 Hz setups, QA-CMIF saves 46%-55% in parameters and 72%-81% in compute over recent baselines with a 1.44 ms 99th-percentile single-thread CPU latency.
BibTeX
@misc{tao2026qa,
  author = {Xinyu Tao and Shuang Huang and Yeyun Cai and Fang Deng},
  title = {{QA-CMIF:Accurate and Lightweight Cross-Modal Latent Imputation and Fusion for Fault-Tolerant sEMG-Inertial Hand Gesture Recognition}},
  year = {2026},
  eprint = {2611.20124},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20124},
}
Figure