An Avatar Robot Overlaid with the 3D Human Model of a Remote Operator

Ravi Tejwani, Chengyuan Ma, Paolo Bonato, H. Harry Asada

TL;DR

This paper introduces an Avatar Robot System that overlays a physical telepresence robot with a 3D Augmented Reality (AR) model of a remote caregiver. By using a novel, high-speed, non-iterative alignment solver called ONIA, the system ensures that when a patient feels the robot's touch, they simultaneously see the caregiver’s virtual arm perfectly aligned with the physical movement, bridging the gap between haptic feedback and visual realism.

What Problem Does This Paper Address?

While telepresence robots are increasingly used in healthcare and rehabilitation, they often suffer from a "trust gap." Patients may find it uncomfortable or even frightening to be touched by a cold, mechanical arm while the human operator is only visible on a 2D screen.

The technical challenge lies in visual-haptic synchronization. Humans and robots have different limb proportions. To make a remote operator appear to be the one physically interacting with a patient, the system must map a 3D human model onto a robot’s frame in real-time. Existing Inverse Kinematics (IK) solvers are often too slow, prone to mathematical "singularities" (where the math breaks down, like when an arm is perfectly straight), or require heavy manual tuning to look "natural."

Methods

The authors developed an end-to-end pipeline involving an ABB YuMi (a collaborative dual-arm robot), a control station, and a Microsoft HoloLens 2 for the AR overlay.

The core contribution is the Optimal Non-Iterative Alignment (ONIA) solver. Unlike standard iterative solvers (like Jacobian DLS or FABRIK) that "hunt" for a solution over multiple steps, ONIA uses a closed-form geometric approach:

  1. Arm Axis Computation: It treats the upper arm and forearm as vectors.
  2. Sequential Alignment: It aligns the shoulder's "swing" (direction) then its "twist" (rotation around the bone axis) to match the robot’s elbow position.
  3. Dynamic Scaling: Because robot and human arms differ in length, ONIA calculates a "stretch factor" for the virtual limbs to ensure the virtual joints (elbow/wrist) land exactly where the physical robot joints are.
  4. Wrist Twist Reactivity: It includes a heuristic to propagate wrist rotation back into the forearm mesh, making the skin deformation look more realistic.

Datasets

The paper does not use a large-scale external dataset. Instead, it evaluates the system using:

  • A 3D Photogrammetry Scan: A high-fidelity scan of a real physical therapist in a "T-pose."
  • Static Pose Benchmark: A set of 12 representative arm poses (e.g., "hands up," "arms in front," "A-pose") used to compare solver accuracy.
  • Emulated Therapy Session: A recorded sequence of "shoulder flexion with elbow extension" therapy to test real-time performance and stability.

Key Insights and Ideas

  • The Necessity of Stretching: The authors argue that for a convincing AR overlay, "anthropomorphic correctness" (keeping limb lengths constant) is less important than "visual overlap." They allow the human model to stretch or shrink to ensure the virtual hand is exactly where the robot hand is.
  • Non-Iterative is King for AR: In AR, latency causes "swimming" (where the virtual object lags behind the real world). Because ONIA is non-iterative, it is an order of magnitude faster than traditional IK, making it ideal for 60Hz+ AR rendering.
  • Singularity-Free Design: By using a deterministic geometric mapping rather than gradient descent, the solver avoids the "glitches" that happen when a robot arm reaches a limit or a perfectly straight configuration.

Key Results

  • Speed: ONIA is significantly faster than Jacobian and FABRIK solvers, which are often 10x more computationally expensive.
  • Visual Accuracy: ONIA achieved the highest Overlay Ratio (a metric introduced by the authors to measure how much of the physical robot is hidden by the virtual model from the patient's perspective).
  • Precision: ONIA ensures zero deviation for the elbow and wrist joints relative to the robot's reference points, whereas iterative solvers often leave small gaps.
  • Robustness: In the emulated therapy session, the FABRIK solver "broke" (hit a singularity) and provided poor alignment for half the session, while ONIA remained stable throughout.

Limitations and Future Work

  • Excessive Scaling: In some extreme poses, the human model stretches by up to 1.5x, which can look unnatural. Future versions might need to balance "perfect overlap" with "anatomical plausibility."
  • Torso Integration: Currently, the solver treats each arm independently. It does not yet account for how the torso should move or twist in coordination with the arms.
  • Hand Gestures: The current system maps a simple 1-degree-of-freedom gripper to a hand gesture but doesn't yet support complex, multi-finger teleoperation.
  • Subjective Validation: The study focused on objective engineering metrics (speed, overlap). The authors acknowledge the need for future user studies to see if patients actually feel more trust and less "terror" when using the system.

Summary updated

BibTeX
@inproceedings{tejwani2023avatar,
  author = {Ravi Tejwani and Chengyuan Ma and Paolo Bonato and H. Harry Asada},
  title = {{An Avatar Robot Overlaid with the 3D Human Model of a Remote Operator}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  doi = {10.1109/iros55552.2023.10341890},
}
Figure