TCP4D: A Unified Tool-Center-Point Tracking and Generation Model for Robot Manipulation

Haosong Peng, Xiangyang Liu, Zhengshen Zhang, Yunfeng Fan, Tianyu Qi, Xin Xia, Yufeng Zhan, Wenchao XuarXiv 2026 · arXiv

TL;DR

TCP4D is a unified model that tracks and generates 6D tool-center-point (TCP) trajectories directly from monocular RGB video. By utilizing an embodiment-agnostic action space, it enables a powerful cross-embodiment data augmentation pipeline that can transform a video of one robot into training data for another, significantly boosting success rates on new hardware without additional data collection.

What Problem Does This Paper Address?

Large-scale video datasets are the lifeblood of modern AI, but in robotics, they come with a major catch: most "in-the-wild" videos lack synchronized action annotations (the precise 6D poses and gripper states). Without these labels, we can't easily use this data to train vision-language-action (VLA) models.

Existing 4D point trackers can follow pixels across frames, but they struggle with the Tool-Center-Point (TCP)—the virtual origin between a robot's gripper fingers. The TCP often resides in free space, meaning its image projection might fall on a background pixel, and point trackers don't provide the 3D orientation (rotation) necessary for actual manipulation. Recovering these "hidden" actions from raw video usually requires complex, robot-specific post-processing that is prone to error.

Methods

The authors introduce TCP4D, a framework that treats tracking (understanding the past) and generation (predicting the future) as two sides of the same coin, powered by a shared 4D scene representation.

Overview of the TCP4D architecture, showing the unified Tracker and Generator modules.

1. The Tracker

Given a monocular video and a simple 2D query (clicking on the gripper in the first frame), the Tracker reconstructs the scene's geometry and recovers the metric TCP poses in SE(3) along with gripper states. It uses a DINOv2-based Transformer backbone to build a unified 4D representation of the video.

2. The Generator

The Generator acts as a manipulation policy. It samples local features from the same 4D representation along the tracked paths and uses a Diffusion Transformer (DiT) to predict future TCP trajectories and gripper states, conditioned on language instructions and robot observations.

3. Cross-Embodiment Augmentation

Perhaps the most innovative part of the paper is the augmentation pipeline. Because TCP poses are embodiment-agnostic (a "move to this coordinate" command works for any robot that can reach it), the authors can:

  1. Recover the TCP motion from a source video (e.g., a Franka Panda).
  2. Use a segmentation model (SAM3) and inpainting to "erase" the original robot.
  3. Render a target robot (e.g., a UR5 or JACO) following that same trajectory via Inverse Kinematics (IK).
The cross-embodiment augmentation pipeline: from source demo to target-robot training data.

Datasets

The model was trained and evaluated on a diverse set of benchmarks:

  • DROID: A massive real-world dataset of Franka Panda robot demonstrations.
  • RoboLab: 100 "in-the-wild" episodes used for testing generalization to unseen tasks.
  • RoboTwin: A simulation benchmark with 48 distinct tasks used to evaluate the Generator's policy performance.
  • Berkeley AUTOLAB UR5: Used to demonstrate the model's ability to adapt to a completely different robot embodiment with minimal data.

Key Insights and Ideas

  • TCP as a Universal Action Space: By focusing on the tool-center-point rather than joint angles, the model creates a "universal language" for manipulation that ignores the specific mechanical details of the robot arm.
  • Unified Tracking and Generation: Sharing the 4D representation between the tracker and generator allows the model to leverage the rich geometric understanding gained during tracking to make better future predictions.
  • Few-Shot Embodiment Adaptation: The model is remarkably efficient. It can adapt to a new robot (like the UR5) with as few as ten demonstrations, reducing the Absolute Trajectory Error (ATE) to just 8.85 mm.
  • Augmentation is a Force Multiplier: Training on a mix of original and "retargeted" augmented data significantly outperforms training on augmented data alone, suggesting that the diversity of visual appearances helps the model learn more robust features.

Key Results

  • Tracking Accuracy: TCP4D achieved state-of-the-art results with an ATE of 7.60 mm on DROID and 16.07 mm on the challenging RoboLab set.
  • Manipulation Success: In the RoboTwin simulation, the TCP4D generator achieved a 69.19% mean success rate, outperforming established models like ACT (29.25%) and DP3 (54.27%).
  • Real-World Impact: On physical robot tasks (placing pens and stacking bowls), TCP4D achieved an 83.33% success rate, significantly higher than the 66.7% achieved by DP3.
  • Augmentation Gains: Joint training on source and augmented demonstrations improved target-robot success by 41.5% compared to training on augmented data only.
Success rates across different tasks and training regimes, highlighting the benefits of joint training.

Limitations and Future Work

While TCP4D is a major step forward, the authors note a few areas for improvement:

  • Embodiment Coverage: Currently, the training data is heavily weighted toward Franka arms and Robotiq grippers. Expanding to more diverse hardware is a priority.
  • Environmental Generalization: While the model performs excellently in "clean" environments, its success rate drops in highly randomized settings (the "clean2random" test). The authors suggest that large-scale pretraining on more diverse 3D geometric data could bridge this gap.
  • Occlusion: The model currently excludes tasks where the TCP is totally occluded from the camera's view, a common challenge in complex manipulation.

Summary updated

Abstract

Robot manipulation videos provide a rich visual experience, yet recovering robot action and gripper-state annotations from RGB observations still remains challenging. Using the embodiment-agnostic tool-center-point (TCP) pose as the action space, we present TCP4D, a unified model in which a shared 4D scene representation both tracks observed TCP motion and generates future TCP trajectories. Given a monocular video and initial 2D TCP queries, the tracker directly recovers metric TCP poses in SE(3) and gripper states alongside scene geometry and camera parameters. The generator samples local features from the same 4D representation along the recovered TCP tracks, then combines them with robot observations and language instructions to predict future TCP states. Our new cross-embodiment augmentation pipeline converts existing RGB videos into demonstrations for alternative embodiments by retargeting the recovered motion and synthesizing the corresponding observations, without recorded source-robot states. Experiments show that the tracker achieves absolute trajectory errors of 7.60 and 16.07 mm on DROID and RoboLab and adapts to a new embodiment with only ten demonstrations. The generator achieves 69.19% mean success on RoboTwin and outperforms well-established policy models on two real-world tasks, while joint training on source and augmented demonstrations improves target-robot success by 41.5% over augmented-only training.
BibTeX
@misc{peng2026tcp,
  author = {Haosong Peng and Xiangyang Liu and Zhengshen Zhang and Yunfeng Fan and Tianyu Qi and Xin Xia and Yufeng Zhan and Wenchao Xu},
  title = {{TCP4D: A Unified Tool-Center-Point Tracking and Generation Model for Robot Manipulation}},
  year = {2026},
  eprint = {2611.23143},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23143},
}
Figure