Language Priors on Rod Stiffness for Real-Time Splatting of Moving Cables

Cables, ropes, wires and plant stems are among the hardest things to reconstruct and track from cameras. They are thin, often narrower than a pixel. They move. And how they move depends on physical properties that cameras cannot see directly: how hard the object is to bend, how hard it is to twist, and how much it weighs per unit length. The best current system that combines 3D rendering with physics for such objects, DLO-Splatting, runs a rope simulator to predict the next shape and then corrects that shape by gradient descent on how well the rendered rope matches the images. It reaches only about 1 Hz, and its physics parameters are hand-picked. This proposal argues that these two facts are linked. When the predicted shape is wrong, the correction step needs many expensive iterations. A good physical prior, obtained by asking a vision-language model what the object is and turning that answer into a calibrated probability distribution over its mechanical constants, should shrink the prediction error enough that a small, fixed number of correction steps suffices. That is the route to real-time rates.

The core idea, in two sentences: a vision-language model looks at a cable or stem and supplies an uncertain prior over its bending stiffness, twisting stiffness and mass per unit length; this prior drives a 1D rod simulator, and its posterior is refined online from tracking residuals. Each frame, a rod-aligned Gaussian rendering model corrects the simulator's prediction with only a few gradient steps, so physical knowledge from language becomes both better tracking under occlusion and lower latency.

Core Claim

Our claim is that for thin deformable structures, inferring physical parameters from semantics is a latency mechanism, not only a realism feature. Consider a predict–correct tracker whose correction step runs $K$ gradient iterations per frame. The number of iterations needed to reach a fixed image-matching error grows with the size of the gap between the simulator's predicted centerline and the true one. That gap is dominated by error in the rod's constants. Better-calibrated constants therefore allow a smaller $K$ at equal accuracy. We will test this directly by measuring iterations-to-convergence against parameter error.

The second claim is that the parameters must be treated as an evolving posterior rather than a one-shot answer. Predictions from vision-language models are known to be noisy and inconsistent. PhysGS fixes this for static objects by accumulating per-view estimates with conjugate Bayesian priors. We instead update the posterior with the evidence that matters for dynamics, which is how far the observed motion departs from the predicted motion.

What Is New Relative to Prior Work

The motivating evidence comes from three papers that each lack two of the three ingredients.

DLO-Splatting pairs a position-based rope model with spherical Gaussians along a centerline. It is slow (about 1 Hz), uses fixed heuristic parameters, and cannot render twist because spheres carry no orientation. Gazebo Plants simulates stems as Cosserat rods, meaning curves that track both position and a material orientation frame so that bending and twisting are modeled, and it runs near real time (0.93 simulation-to-real-time ratio on an RTX 3090). However, its parameters are tuned by hand against pull tests, and it has no image-based reconstruction at all. PhysGS and PUGS infer density, friction, hardness and Young's modulus zero-shot from vision-language models on Gaussian splats. Both are restricted to static, volumetric household objects on ABO-500. PUGS uses stiffness only to choose a grasp aperture, never to drive motion.

Nobody in the file maps semantics to the 1D rod constants: bending rigidity $EI$, torsional rigidity $GJ$, axial stiffness $EA$, and mass per unit length. Nobody uses such a mapping to make tracking cheaper.

The method is assembled from reusable parts. We reuse Cosserat position-based dynamics from Gazebo Plants as the predictor. We reuse the predict–correct structure and the gsplat-style differentiable rasterization from DLO-Splatting as the corrector. We reuse PhysGS's Normal-Inverse-Gamma conjugate prior, a closed-form distribution over an unknown mean and variance, as the parameter belief.

Two papers constrain the design. Fracture-GS and GASP show that volumetric particle physics on Gaussians costs roughly 6–10 ms per frame or tens of seconds per sequence even with clustering. This argues for a reduced 1D model rather than solid simulation for cables. GeoGaussian flattens Gaussians into surface patches. That is the wrong shape prior for a cylinder, which is curved around its axis and straight along it.

Research Program

The representation is a Cosserat rod: $N$ centerline nodes with positions and quaternion frames, simulated with position-based constraints for stretch, shear, bend and twist. Each segment carries a small ring of anisotropic Gaussians whose long axis follows the segment tangent and whose rotation follows the segment's material frame. Twist then appears visibly in surface texture, which spherical chains cannot show. Cross-section radius is a per-node learnable parameter.

The background is reconstructed once as a static splat and frozen. This deliberately sidesteps the failure analyzed for RobustSplat, whose coarse-resolution masking can confuse sub-pixel thin structures with transient noise. We instead obtain an initial rod mask by prompting a segmenter with the vision-language model's detection. The rod is then initialized by fitting its centerline to the first few multi-view frames.

The semantic-to-physical step works as follows. We prompt the vision-language model with crops of the object for a material and category label ("braided nylon rope", "PVC-insulated power cable", "green tomato stem"). We also ask for ranges of Young's modulus, density and typical diameter. These are converted into rod constants under a solid circular cross-section, which gives

$$EI = \frac{\pi E d^4}{64}, \qquad \lambda = \frac{\pi \rho d^2}{4},$$

where $E$ is Young's modulus, $d$ the diameter, $\rho$ density and $\lambda$ mass per unit length.

The fourth-power dependence on $d$ is a central hazard. A thin cable's diameter is poorly observed, so a 30% diameter error becomes almost a factor of three in bending stiffness. We therefore place the prior on $\log EI$, $\log GJ$ and $\log \lambda$ directly. Its variance is inflated by both the range the model reports and the image-based diameter uncertainty. For composite cables (braid, insulation over copper) we let the model's stated range dominate the solid-section formula, since that formula does not hold for them. We also map these constants into the simulator's compliance parameters through its physical-unit formulation. This addresses the calibration gap noted in the file between real SI constants and discrete simulator stiffness. We validate that mapping separately on a cantilever-sag test.

Tracking runs per frame. First the simulator predicts the next state using the current posterior mean. Then $K$ gradient steps on node positions minimize the rendering loss on the rod's pixels, together with a term keeping the rod near the predicted shape. The correction left over after these steps is the innovation. Because it carries information about the parameters, it updates the posterior. We use a low-dimensional fit: every $T$ frames, we re-simulate the last window under a few sampled parameter values and update the Gaussian belief over log-constants toward values that would have reduced the corrections. This yields a sequential estimate in which the prior's influence decays as motion evidence accumulates.

The latency budget is set by fixing $K$ rather than iterating to convergence. Rendering a few thousand rod Gaussians over a frozen background, plus a single-rod position-based solve, should fit a 30 Hz budget. Measuring this on the same GPU class as the cited systems is part of the work.

Baselines

The strongest competitor is DLO-Splatting. It is the only system in the file that tracks a thin deforming object with physics prediction plus Gaussian-rendering correction, and it outperforms the vision-only tracker TrackDLO on a 12-second knot-tying "Cross Move" sequence. Its reported operating point is an update rate of approximately 1 Hz. For its tracking error, no reported result was found on Lacuna. We will re-implement or run it on our sequences with its hand-tuned parameters. We will report it twice: at its native iteration count, and at our fixed per-frame iteration budget $K$. Beating it means matching its tip and centerline error at 30 Hz, or achieving lower error at equal update rate.

The kinematic reference is 4D Gaussian Splatting. It reports 34.05 dB average PSNR and 82 FPS at 800×800 on D-NeRF with an RTX 3090. Frequency-aware Dynamic Gaussian Splatting reports 42.76 PSNR on D-NeRF and is the strongest kinematic method for sharp moving detail. Both will be trained on the same multi-view sequences with their default schedules. Because they are offline optimizers, we compare them mainly on rendering fidelity and on open-loop prediction, which they cannot do.

The naive reference point is our own pipeline with a generic default rod (a fixed mid-range stiffness and density for every object) and no posterior update. The key ablations isolate the semantic prior. They are: a default prior with the online posterior update, to test whether motion evidence alone suffices; the language prior as a fixed point estimate without uncertainty, to test whether the Bayesian treatment matters as PhysGS found for static properties; and an oracle using parameters measured in the lab, which upper-bounds what any prior could deliver. A rendering ablation swaps our oriented Gaussians for DLO-Splatting's spheres to isolate the contribution of the rod-frame primitive. All variants share data, GPU, and iteration budget.

Why not just…?

Why not just identify the stiffness from the observed motion and skip the language model? This is the most serious threat, and it may be partly right. Given enough informative motion, fitting the constants from video should converge regardless of the starting point, and the prior then washes out. The language prior can matter only in three situations: the cold-start window before informative motion has been seen; quasi-static or slow manipulations where bending stiffness is weakly identifiable; and prediction before first contact, as in planning a grasp. The default-prior-plus-update ablation settles this. We will plot parameter error and tracking error against elapsed time. If the two curves merge within about a second on most objects, we will narrow the claim to cold-start and pre-contact prediction rather than argue around it.

Why not just run DLO-Splatting with fewer gradient steps, or on a faster GPU? Hardware helps every method equally. Cutting iterations without a better predictor should raise error, because the correction step must close a larger gap. The fixed-$K$ comparison against DLO-Splatting tests exactly this. The plot of iterations-to-convergence against parameter error, using oracle, language-prior and default parameters, directly tests the causal link we claim. If error at fixed $K$ is insensitive to parameter quality, the latency argument fails.

Why not just use a kinematic 4D deformation field, which already renders in real time? 4D-GS predicts independent per-Gaussian offsets with no length or bending constraint. Frequency-aware Dynamic GS documents that sharp detail and fast motion conflict in such fields. More fundamentally, these fields are fitted offline to a whole sequence. They cannot predict where an occluded cable will be, and they cannot be run online as a tracker. The open-loop occlusion test below makes this concrete.

Why not just prompt the vision-language model once and use the number? PhysGS reports that Bayesian accumulation reduced Shore hardness error by 61.2% and friction error by 18.1% relative to the deterministic NeRF2Physics baseline on ABO-500. That is evidence that point estimates are unreliable even for static properties. The $d^4$ sensitivity of bending stiffness makes overconfidence worse for rods. The point-estimate ablation tests whether this carries over.

Proposed Experiments

No benchmark in the file contains thin rods with ground-truth mechanical constants. PointWire and PointVessel provide thin-tube geometry only, and D-NeRF contains no cables. We therefore build two evaluation sets.

The synthetic set renders rods simulated with the Gazebo Plants Cosserat formulation, using known, varied constants and textures, from multiple cameras. This gives exact centerline and parameter ground truth.

The real set covers about twenty physical objects: nylon rope, braided rope, USB and power cables, solid copper wire, rubber tubing, and plant stems. Each is measured offline for bending stiffness (cantilever deflection) and mass per unit length. Each is manipulated by a robot arm under a multi-camera rig, with sparse markers for ground-truth centerline points. We also replay DLO-Splatting's "Cross Move" knot-tying protocol for direct comparability.

The metrics are: tip and centerline error against markers; rendering PSNR and LPIPS restricted to rod pixels; update rate in Hz with per-component timing; absolute log-error of $EI$ and mass per unit length over time; and open-loop prediction error.

The decisive check is occlusion. We blank all cameras for 0.5–2 seconds mid-manipulation, while gripper states remain known, and measure how far each method's predicted centerline drifts. A tracker with correct constants should drift little. A default-parameter tracker should drift substantially. Kinematic fields cannot produce a prediction at all.

The second decisive check is the iterations-versus-parameter-error curve described above. The third is the time-to-merge comparison between language-prior and default-prior posteriors. It determines honestly whether semantic knowledge earns its place over pure system identification, or only shortens the cold start.