TL;DR
The paper introduces HMR-1, a hierarchical robotic framework that combines Multimodal Large Language Models (MLLMs) with low-level kinematic control to perform precise, instruction-driven acupoint massages. To power this system, the authors also release MedMassage-12K, the first large-scale multimodal dataset specifically designed for embodied healthcare massage tasks.
What Problem Does This Paper Address?
While Embodied AI has made massive strides in robotic manipulation (like grabbing objects or opening drawers), its application in healthcare—specifically physical therapy and rehabilitation—remains severely limited.
The authors identify two major bottlenecks:
- The Scarcity of Data: There are no open-source, large-scale multimodal datasets mapping natural language instructions to physical acupoints under diverse environmental conditions.
- The Grounding Gap: Traditional object detectors (like YOLO or Faster R-CNN) can find static objects but cannot parse complex, context-dependent human instructions (e.g., "Locate the Zusanli acupoint and apply moderate pressure"). Conversely, state-of-the-art commercial MLLMs excel at understanding language but lack the fine-grained spatial awareness required to guide a robot arm safely to a precise millimeter-scale location on a human body.
Methods
The authors tackle this challenge using a decoupled, hierarchical framework divided into two core components:
1. High-Level Grounding Module (HLGM)
This module handles semantic interpretation and 2D localization. It takes a natural language command and an RGB image as inputs.
- Architecture: It leverages a fine-tuned Qwen-VL model. Visual features are extracted using an OpenCLIP ViT-bigG encoder and passed through a single-layer cross-attention vision-language adaptor.
- Output: Instead of just text, the language model decoder (Qwen-7B) is trained to output normalized 2D bounding box coordinates demarcating the target acupoint.
2. Low-Level Control Module (LLCM)
Once the 2D coordinates are established, the system transitions from pixels to physical space:
- 3D Mapping: Using depth data from an RGB-D camera and its intrinsic/extrinsic parameters, the 2D pixel coordinates are projected into a 3D point cloud.
- Pose Estimation: The system applies the RANSAC algorithm for plane fitting to determine the surface's normal vector. By utilizing a "vertical tapping principle," it deduces the exact 6-DOF (Degrees of Freedom) end-effector pose (position and orientation).
- Trajectory Generation: Inverse Kinematics (IK) algorithms calculate the joint configurations, and polynomial fitting combined with spline interpolation ensures a smooth, collision-free path to prevent abrupt acceleration changes against the patient.
Datasets
The paper introduces MedMassage-12K, a novel dataset built using medical training mannequins marked with acupoints.
- Raw Data: Initially consists of 1,030 acupoint images and 15,092 annotated question-and-answer (QA) pairs covering 60 distinct massage points.
- Augmented Data: To ensure the model generalizes across different body positions, heights, and environments, the authors applied geometric transformations (random cropping and rotations).
- Final Count: 12,190 images and 174,177 QA pairs captured under three distinct lighting conditions: Natural, Dim, and Bright Light.
Key Insights and Ideas
- Zero-Shot MLLMs Fail at Medical Grounding: One of the most striking findings is that commercial giants like GPT-4o and Qwen-VL-Max completely fail at this task out of the box, achieving near 0% success rates. General-purpose vision-language models simply do not possess the domain-specific spatial precision required for healthcare robotics without targeted fine-tuning.
- Data Augmentation is Paramount: Simply scaling up data isn't enough; environmental diversity matters. Introducing geometric and lighting variations via data augmentation improved the model's grounding success rate at an Intersection over Union (IoU) threshold of 0.5 by over 32%.
- The Power of Hierarchical Decoupling: Trying to map text directly to robot joint torques is incredibly difficult. By separating the task into Language $\rightarrow$ Bounding Box (High-Level) and Bounding Box $\rightarrow$ 3D Trajectory (Low-Level), the framework remains interpretable, safe, and easier to optimize.
Key Results
The authors benchmarked their fine-tuned model against leading commercial models on the MedMassage-12K test set, evaluating the Grounding Success Rate across different IoU thresholds:
| Model | IoU = 0.3 | IoU = 0.5 | IoU = 0.75 |
|---|---|---|---|
| Qwen-VL-Max | 0.12% | 0% | 0% |
| GPT-4o | 0.59% | 0.07% | 0% |
| Our Model | 87.60% | 81.42% | 67.77% |
The framework was also validated in physical experiments using a Franka Panda Robot arm equipped with a massage ball actuator and a RealSense D455 camera, successfully executing accurate acupoint localization and trajectory tracking in real-world environments.
Limitations and Future Work
While the results are promising, a few limitations are apparent from the scope of the paper:
- Mannequin vs. Human Subjects: The dataset and physical validation were conducted entirely on rigid medical dummies. Real human bodies present challenges like skin deformation, respiration movements, and varying body mass indexes (BMIs), which will require dynamic visual tracking.
- Lack of Force Feedback: The paper focuses heavily on spatial trajectory planning but does not deeply explore haptic or compliance control (force feedback), which is critical to ensure a robot does not press too hard against human tissue.
Summary updated