HMR-1: Hierarchical Massage Robot with Vision-Language-Model for Embodied Healthcare

Rongtao Xu, Mingming Yu, Xiaofeng Han, Yu Zhang, Kaiyi Hu, Zhe Feng, Zenghuang Fu, Changwei Wang, Weiliang Meng, Xiaopeng ZhangarXiv 2026 · arXiv

TL;DR

The paper introduces HMR-1, a hierarchical robotic framework that combines Multimodal Large Language Models (MLLMs) with low-level kinematic control to perform precise, instruction-driven acupoint massages. To power this system, the authors also release MedMassage-12K, the first large-scale multimodal dataset specifically designed for embodied healthcare massage tasks.

What Problem Does This Paper Address?

While Embodied AI has made massive strides in robotic manipulation (like grabbing objects or opening drawers), its application in healthcare—specifically physical therapy and rehabilitation—remains severely limited.

The authors identify two major bottlenecks:

  1. The Scarcity of Data: There are no open-source, large-scale multimodal datasets mapping natural language instructions to physical acupoints under diverse environmental conditions.
  2. The Grounding Gap: Traditional object detectors (like YOLO or Faster R-CNN) can find static objects but cannot parse complex, context-dependent human instructions (e.g., "Locate the Zusanli acupoint and apply moderate pressure"). Conversely, state-of-the-art commercial MLLMs excel at understanding language but lack the fine-grained spatial awareness required to guide a robot arm safely to a precise millimeter-scale location on a human body.

Methods

The authors tackle this challenge using a decoupled, hierarchical framework divided into two core components:

Figure 1: The proposed HMR framework. Given a textual instruction and an RGB-D observation, the model predicts the 6-DOF end-effector pose for robotic massage.

1. High-Level Grounding Module (HLGM)

This module handles semantic interpretation and 2D localization. It takes a natural language command and an RGB image as inputs.

  • Architecture: It leverages a fine-tuned Qwen-VL model. Visual features are extracted using an OpenCLIP ViT-bigG encoder and passed through a single-layer cross-attention vision-language adaptor.
  • Output: Instead of just text, the language model decoder (Qwen-7B) is trained to output normalized 2D bounding box coordinates demarcating the target acupoint.

2. Low-Level Control Module (LLCM)

Once the 2D coordinates are established, the system transitions from pixels to physical space:

  • 3D Mapping: Using depth data from an RGB-D camera and its intrinsic/extrinsic parameters, the 2D pixel coordinates are projected into a 3D point cloud.
  • Pose Estimation: The system applies the RANSAC algorithm for plane fitting to determine the surface's normal vector. By utilizing a "vertical tapping principle," it deduces the exact 6-DOF (Degrees of Freedom) end-effector pose (position and orientation).
  • Trajectory Generation: Inverse Kinematics (IK) algorithms calculate the joint configurations, and polynomial fitting combined with spline interpolation ensures a smooth, collision-free path to prevent abrupt acceleration changes against the patient.

Datasets

The paper introduces MedMassage-12K, a novel dataset built using medical training mannequins marked with acupoints.

Figure 2: The bar chart on the left shows the distribution of acupoints in the test set and training set, while the right side displays images of the mannequin with acupoints under different lighting and background conditions.
  • Raw Data: Initially consists of 1,030 acupoint images and 15,092 annotated question-and-answer (QA) pairs covering 60 distinct massage points.
  • Augmented Data: To ensure the model generalizes across different body positions, heights, and environments, the authors applied geometric transformations (random cropping and rotations).
  • Final Count: 12,190 images and 174,177 QA pairs captured under three distinct lighting conditions: Natural, Dim, and Bright Light.

Key Insights and Ideas

  • Zero-Shot MLLMs Fail at Medical Grounding: One of the most striking findings is that commercial giants like GPT-4o and Qwen-VL-Max completely fail at this task out of the box, achieving near 0% success rates. General-purpose vision-language models simply do not possess the domain-specific spatial precision required for healthcare robotics without targeted fine-tuning.
  • Data Augmentation is Paramount: Simply scaling up data isn't enough; environmental diversity matters. Introducing geometric and lighting variations via data augmentation improved the model's grounding success rate at an Intersection over Union (IoU) threshold of 0.5 by over 32%.
  • The Power of Hierarchical Decoupling: Trying to map text directly to robot joint torques is incredibly difficult. By separating the task into Language $\rightarrow$ Bounding Box (High-Level) and Bounding Box $\rightarrow$ 3D Trajectory (Low-Level), the framework remains interpretable, safe, and easier to optimize.

Key Results

The authors benchmarked their fine-tuned model against leading commercial models on the MedMassage-12K test set, evaluating the Grounding Success Rate across different IoU thresholds:

Model IoU = 0.3 IoU = 0.5 IoU = 0.75
Qwen-VL-Max 0.12% 0% 0%
GPT-4o 0.59% 0.07% 0%
Our Model 87.60% 81.42% 67.77%

The framework was also validated in physical experiments using a Franka Panda Robot arm equipped with a massage ball actuator and a RealSense D455 camera, successfully executing accurate acupoint localization and trajectory tracking in real-world environments.

Limitations and Future Work

While the results are promising, a few limitations are apparent from the scope of the paper:

  • Mannequin vs. Human Subjects: The dataset and physical validation were conducted entirely on rigid medical dummies. Real human bodies present challenges like skin deformation, respiration movements, and varying body mass indexes (BMIs), which will require dynamic visual tracking.
  • Lack of Force Feedback: The paper focuses heavily on spatial trajectory planning but does not deeply explore haptic or compliance control (force feedback), which is critical to ensure a robot does not press too hard against human tissue.

Summary updated

Abstract

The rapid advancement of Embodied Intelligence has opened transformative opportunities in healthcare, particularly in physical therapy and rehabilitation. However, critical challenges remain in developing robust embodied healthcare solutions, such as the lack of standardized evaluation benchmarks and the scarcity of open-source multimodal acupoint massage datasets. To address these gaps, we construct MedMassage-12K - a multimodal dataset containing 12,190 images with 174,177 QA pairs, covering diverse lighting conditions and backgrounds. Furthermore, we propose a hierarchical embodied massage framework, which includes a high-level acupoint grounding module and a low-level control module. The high-level acupoint grounding module uses multimodal large language models to understand human language and identify acupoint locations, while the low-level control module provides the planned trajectory. Based on this, we evaluate existing MLLMs and establish a benchmark for embodied massage tasks. Additionally, we fine-tune the Qwen-VL model, demonstrating the framework's effectiveness. Physical experiments further confirm the practical applicability of the framework.Our dataset and code are publicly available at https://github.com/Xiaofeng-Han-Res/HMR-1.
BibTeX
@misc{xu2026hmr,
  author = {Rongtao Xu and Mingming Yu and Xiaofeng Han and Yu Zhang and Kaiyi Hu and Zhe Feng and Zenghuang Fu and Changwei Wang and Weiliang Meng and Xiaopeng Zhang},
  title = {{HMR-1: Hierarchical Massage Robot with Vision-Language-Model for Embodied Healthcare}},
  year = {2026},
  eprint = {2603.08817},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.08817},
}
Figure