WorldString: Actionable World Representation

Kunqi Xu, Jitao Li, Jianglong Ye, Tianshu Tang, Isabella Liu, Sifei Liu, Xueyan ZouarXiv 2026 · arXiv

TL;DR

WorldString is a unified, fully differentiable neural architecture that models the dynamic state manifolds of articulated, skinned, and soft objects directly from 3D point clouds or RGB-D video. By mathematically relaxing classical physical kinematics into a keypoint-driven transformer framework, it serves as a highly controllable digital twin and a foundational building block for physical world models.

What Problem Does This Paper Address?

Building AI agents capable of interacting seamlessly with the physical world requires robust "physical world models." Current approaches generally fall into two camps: top-down generative video models (which lack strict 3D consistency and precise controllability) and bottom-up dynamic scene reconstruction methods (which struggle with contact-rich interactions and generalization).

Crucially, existing methods treat different object types—such as rigid robot arms, skinned animal bodies, and highly deformable soft cloths—using completely separate mathematical frameworks (e.g., Forward Kinematics vs. Linear Blend Skinning vs. Finite Element Method Jacobians). This paper addresses the lack of a unified, principled representation capable of explicitly modeling all these diverse object deformation regimes under a single, controllable neural architecture.

Position of object representation under the scope of physical world model.

Methods

The core insight of WorldString is that classical physical formulations share a common underlying structure: they can all be expressed as a convex combination of keypoint-induced updates. This allows the authors to use cross-attention as a learned, generalized relaxation of these physical priors.

The WorldString architecture consists of a two-stage transformer pipeline:

  1. State Transformer ($\Phi_s$): Takes learnable canonical base embeddings ($\omega_0$) representing the object's rest state and uses cross-attention to condition them on a sparse set of dynamic structural keypoints ($K$). This injects localized keypoint constraints into the canonical geometry.
  2. Object Transformer ($\Phi_o$): Applies self-attention over the intermediate state embeddings to propagate localized deformations globally and enforce structural coherence across the entire object manifold, outputting structured embeddings ($Z_{obj}$).
  3. Voxel Transformer ($\Phi_v$): To map these latent embeddings back into interpretable Cartesian space, this module cross-attends continuous 3D spatial queries ($Q(x)$) with $Z_{obj}$ to predict a continuous occupancy field ($O(x)$).
WorldString model pipeline showing the cascaded transformers.

For real-world deployment, the authors also introduce a data acquisition pipeline that uses Grounded-SAM2 for object segmentation, CoTracker for dense pixel tracking, and unprojects 2D trajectories into 3D point clouds. Structural anchors are sampled via Farthest Point Sampling (FPS) and tracked over time to serve as the keypoint inputs.

Datasets

The model was evaluated across a diverse set of benchmarks:

  • Rigid Shapes: Standard complex topologies including the Utah Teapot, Stanford Bunny, Armadillo, and Lucy.
  • Articulated Objects: Robotic and furniture datasets including Xhand, Airbot Play, and two IKEA Cabinets.
  • Skinned Objects: Parametric human (SMPL) and animal (SMAL, including horse and hippo) models.
  • Soft Bodies: Real-world deformable sequences from the PhysTwin dataset, featuring a doll, a cloth, and a short rope.

Key Insights and Ideas

  • Unified Physical Relaxation: The mathematical proof demonstrating that Forward Kinematics (FK), Linear Blend Skinning (LBS), and soft-body Jacobians can all be relaxed into a unified residual attention mechanism is a major conceptual leap. It bridges the gap between rigid physics priors and flexible neural implicit fields.
  • Emergent Part Specialization: Without any explicit part-level supervision, the latent query tokens automatically learn to specialize in specific, localized segments of the object's canonical geometry (e.g., always tracking the thumb or a specific limb across highly varied poses).
  • Dual Completion Capabilities: When trained on noisy real-world data, the model exhibits two emergent robustness behaviors: structural completion (reconstructing unobserved geometries caused by self-occlusion) and material completion (filling in artificial "holes" caused by sparse RGB-D sensor data to reconstruct a continuous, solid fabric).

Key Results

  • Articulated Objects: WorldString significantly outperforms specialized baselines like Dr. Robot. On a complex robot hand, WorldString achieved an Intersection over Union (IoU) of 90.28 compared to Dr. Robot's 28.53, completely avoiding the noisy surfaces and hollow artifacts typical of discrete Gaussian kernel methods.
  • Skinning & Soft Bodies: It consistently outperformed Neural Shape Deformation Priors (NSDP) across human and animal categories (e.g., achieving a 92.40 IoU on the Hippo skeleton vs. NSDP's 86.82). For highly non-linear soft bodies like a deformable doll, it achieved an IoU of 82.80.
  • Generality vs. Specialization: When compared against HALO (a model highly specialized for human hands), WorldString matched its performance within a narrow margin (96.24 vs 96.62 IoU) while maintaining the massive advantage of being completely topology-agnostic and applicable to any object class.

Limitations and Future Work

  • Network Capacity Threshold: The ablation studies revealed that simply increasing network depth (e.g., adding more attention layers) or expanding the hidden dimension size does not monotonically improve results. Instead, pushing the model past an optimal capacity threshold introduces unnecessary complexity and causes subtle overfitting to specific training configurations.
  • Future Work: While the architecture provides a fully differentiable spatial representation of object states, the authors note that its seamless differentiability is explicitly designed to enable future integration with policy learning (reinforcement learning) and neural dynamics for full forward-predictive physical simulation.

Summary updated

Abstract

Inspired by the emergent behaviors in large language models that generalized human intelligence, the research community is pursuing similar emergent capabilities within world models, with a emphasis on modeling the physical world. Within the scope of physical world model, objects are the fundamental primitives that constitute physical reality. From humans to computers, nearly everything we interact with is an object. These objects are rarely static; they are actionable entities with varying states determined by their intrinsic properties. While current methods approach object action states either via video generation or dynamic scene reconstruction, none explicitly model this basic element in a unified, principled way to build an actionable object representation. We propose WorldString, a neural architecture capable of modeling the state manifold of real-world objects by learning directly from point clouds or RGB-D video streams. Serving as a versatile digital twin, it acts as a foundational building block for physical world models; thus, we name it WorldString. Sweetly, its fully differentiable structure seamlessly enables future integration with policy learning and neural dynamics.
BibTeX
@misc{xu2026worldstring,
  author = {Kunqi Xu and Jitao Li and Jianglong Ye and Tianshu Tang and Isabella Liu and Sifei Liu and Xueyan Zou},
  title = {{WorldString: Actionable World Representation}},
  year = {2026},
  eprint = {2605.18743},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.18743},
}
Figure