mmTherm: Absolute 3D Human Pose Estimation from Low-Resolution Thermal Images and mmWave Radar

Zhaoxin Chang, Badii Jouaber, Daqing ZhangarXiv 2026 · arXiv

TL;DR

mmTherm is a lightweight framework that achieves state-of-the-art absolute 3D human pose estimation by explicitly fusing millimeter-wave (mmWave) radar with low-resolution thermal imaging. By leveraging the complementary geometric strengths of both sensors—radar for range and thermal for direction—it reduces cross-user localization error by 38% while using 8.5x fewer parameters than previous fusion models.

What Problem Does This Paper Address?

Estimating a person’s exact 3D pose and global position (absolute pose estimation) is critical for smart homes, healthcare, and VR. While RGB cameras are the standard, they raise significant privacy concerns and fail in low-light conditions.

Millimeter-wave (mmWave) radar is a promising, privacy-preserving alternative. However, radar point clouds are notoriously "sparse and noisy." Signals bounce off different body parts depending on orientation, often leaving limbs completely "invisible" to the sensor. Furthermore, multipath reflections create "ghost" points that look like human returns but are actually clutter.

Previous attempts to solve this by fusing radar with thermal imaging relied on "implicit feature-level fusion"—essentially letting a neural network figure out the relationship between the two. The authors of this paper discovered a major flaw: these models fail to generalize. While they perform well on users they were trained on, their accuracy plummets on new subjects because they learn dataset-specific quirks rather than the underlying physical geometry of human movement.

Figure 1. Complementary geometry for absolute 3D HPE. (a) Radar provides range information through sparse and noisy point clouds. (b) Thermal images provide coarse body structure and direction; orange planes illustrate depth ambiguity. (c) mmTherm explicitly combines thermal direction and radar range evidence for pose reconstruction.

Methods

The authors introduce mmTherm, a two-stage framework designed to explicitly exploit the geometric properties of each modality.

  1. Stage 1: Geometry-Guided Root Localization:
    • The Thermal Branch predicts the person's direction and a 2D bounding box from a low-res (32x24) thermal array.
  2. The Radar Branch uses a "voting" mechanism where individual points propose a 3D root (pelvis) location.
  3. A Fusion Head then uses the thermal-derived direction to "correct" the radar's 3D proposal, ensuring the global position is physically plausible.
  4. Stage 2: Root-Conditioned Pose Reconstruction:
    • Using the localized root, the system crops the thermal image and selects nearby radar points.
  5. Thermal-Guided Radar Attention: This is a key innovation. The model uses joint directions inferred from thermal images to "query" the radar points, associating specific points with specific joints (e.g., "find radar points in the direction of the left wrist").
  6. Initialization & Refinement: An initial pose is built by combining thermal directions with radar-derived ranges. This "rough draft" is then polished by a lightweight spatiotemporal Transformer that considers joint relationships over time.
Figure 2. Overview of mmTherm. Stage 1 combines thermal geometry with radar observations to localize the root. Using the predicted root and body box, Stage 2 estimates joint directions from thermal images and joint ranges through thermal-guided radar attention. These estimates form an initial 3D pose, which is then refined by a spatiotemporal network.

Datasets

The model was evaluated on OctoNet, a large-scale multimodal dataset. OctoNet features synchronized data from five sensor nodes (each with a 32x24 thermal array and mmWave radar) and high-fidelity OptiTrack 3D ground truth. The researchers specifically focused on cross-user evaluation, training on one set of people and testing on entirely different individuals to prove the model's real-world robustness.

Key Insights and Ideas

  • Explicit > Implicit Fusion: The most significant "aha moment" is that for cross-user generalization, the model shouldn't just "mix" features. It needs to be forced to respect the physics of the sensors. By using thermal data specifically for direction and radar for range, the model remains grounded in physical reality even when seeing a new body type.
  • Privacy-Resolution Sweet Spot: The use of ultra-low-resolution (32x24) thermal images is a deliberate choice. It’s high enough to provide a "spatial prior" (where the body is) but low enough to protect privacy and keep hardware costs down.
  • Handling "Invisible" Joints: Because radar often misses extremities, the thermal-guided attention allows the model to "hallucinate" missing joints more accurately by using the thermal heat signature as a guide for where those limbs should be.

Key Results

mmTherm significantly outperforms both radar-only and previous fusion-based baselines:

  • Superior Generalization: In cross-user tests, it reduced absolute joint error by 33% and root localization error by 38% compared to the strongest baselines (like MIRaSeg).
  • Extreme Efficiency: The model is remarkably lightweight, requiring only 1.37M parameters (8.5x fewer than MIRaSeg) and achieving 2.4x higher throughput during inference.
  • Robustness to Sparsity: Even when 90% of the radar points were randomly removed, mmTherm maintained lower error rates than other models with full data.
  • Qualitative Improvement: It shows much better performance on dynamic, complex movements like jumping jacks and jump rope, where radar-only methods often lose track of limbs.
Qualitative comparison showing mmTherm's superior accuracy on unseen users across various activities.

Limitations and Future Work

While highly effective, the current version of mmTherm is optimized for single-person scenarios. In crowded environments, associating specific thermal heat signatures with specific radar clusters becomes a much harder "assignment" problem.

Future work could involve extending the framework to multi-person tracking and exploring how to handle severe occlusions (e.g., a person behind furniture) where even thermal signatures might be partially blocked. Additionally, the authors suggest that the geometry-guided approach could be adapted for other "asymmetric" sensor pairs, such as combining low-res cameras with ultrasound.

Summary updated

Abstract

Absolute 3D human pose estimation (HPE) aims to recover both a person's body pose and their global position in 3D space. Privacy-preserving millimeter-wave (mmWave) radar provides distance and direction measurement capabilities, but its sparse point clouds make HPE under-constrained. Low-resolution thermal imaging can provide complementary observations of body direction and structure. However, our evaluation shows that a strong feature-level fusion baseline loses much of its localization and HPE advantage when generalizing to unseen subjects, with radial localization becoming less accurate than a radar-only baseline. Motivated by this finding, we introduce mmTherm, a lightweight framework that explicitly incorporates the complementary geometric properties of thermal imaging and mmWave radar. It combines body structure and joint directions estimated from thermal images with range information from radar to localize the body and reconstruct its 3D pose. Experiments on OctoNet show that mmTherm reduces cross-user absolute joint error by 33% and root localization error by 38% relative to the strongest corresponding baselines, while requiring substantially fewer parameters and less computation at inference.
BibTeX
@misc{chang2026mmtherm,
  author = {Zhaoxin Chang and Badii Jouaber and Daqing Zhang},
  title = {{mmTherm: Absolute 3D Human Pose Estimation from Low-Resolution Thermal Images and mmWave Radar}},
  year = {2026},
  eprint = {2611.23477},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23477},
}
Figure