A soft continuum robot is an arm made of flexible material that bends continuously along its length, with no discrete joints. Robot-learning policies for these arms are currently judged by a single success rate per task. The research file contains a pointed warning about that practice. In the ManiSoft simulator, learned policies did not use the arm's flexibility at all. They commanded straight, rigid-looking paths and collided whenever the scene required bending around something (ManiSoft). If that is the typical failure, then an average success rate mixes two kinds of episode. Some are easy, where a straight reach works. Others can only be solved by curving the body. The average can therefore reward a policy that has learned to be a slightly worse rigid arm.
This note proposes a way to measure, for each benchmark episode, how much the arm must bend to succeed. We call this quantity compliance demand. It is computed from the physics of the expert demonstration and the scene geometry, never from the policy being tested. Episodes are then split into quartiles of compliance demand, and success is reported per quartile. Next to the learned policies we run a deliberately dumb baseline, a scripted straight-line reach that ignores obstacles. The prediction is concrete. On low-demand episodes the dumb baseline will roughly match learned policies. On high-demand episodes every current policy will be near zero. The aggregate score will hide both facts.
Core Claim
Aggregate success on soft-continuum manipulation benchmarks conceals a "rigid shortcut regime." This is a band of episodes solvable without using compliance. It accounts for most of the reported success, and within it a training-free straight-line baseline is competitive. A physics-derived compliance-demand score, independent of both the policy and the success metric, exposes this regime. It also predicts per-episode failure better than the scene labels benchmarks already provide, namely clean versus cluttered scenes, obstacle count, and object category.
The template comes from Aggregate Metrics Hide Shortcut Regimes. That paper split COIL-100 objects into quartiles by a view-change complexity score. It found that a trivial copy-the-input baseline beat nearest-neighbour retrieval on the simplest quartile (VGG distance 0.157 vs 0.353) and lost badly on the hardest (0.520 vs 0.288). Averaged over all quartiles, the two methods looked comparable. That paper supplies the method. It also names its own weakness: its complexity score is a VGG distance, and the model is evaluated with the same perceptual family, which risks circularity. Our design avoids this by construction. Compliance demand is a mechanical quantity, while success is a binary task outcome. The broader motivation comes from the disaggregated-reporting literature. Rethink reporting of evaluation results in AI shows single percentages routinely hide unsafe subgroups and reports that only 4% of surveyed researchers give instance-level results. Coarse-versus-fine washout effects appear in Coarse race data conceals disparities and HERB. None of these works touches robotics, where the natural axis of difficulty is physical rather than demographic or perceptual.
Measuring Compliance Demand
ManiSoft simulates the arm as a Cosserat rod, a one-dimensional model of a slender body that can bend, twist, shear, and stretch. It runs this model in Elastica. Because of this, the full backbone state of every expert demonstration is available. The primary score for an episode is the peak elastic bending-and-twisting energy along the expert trajectory, normalized by the energy of an unobstructed reach to the same target:
$$D = \frac{\max_t \int_0^L \left( B\,\kappa(s,t)^2 + T\,\tau(s,t)^2 \right) ds}{E_{\text{free}}}$$
The symbols are as follows. $s$ is the arc-length position along an arm of length $L$, and $t$ is time. $\kappa$ and $\tau$ are the backbone's curvature and twist. $B$ and $T$ are the bending and torsional stiffnesses. $E_{\text{free}}$ is the peak energy of a straight reach in an empty scene.
A value of $D$ near 1 means the task needs no more bending than a free reach. Large values mean the arm must wrap around obstacles.
The expert demonstration is only one possible solution, so we compute a second, demonstration-free score as a robustness check. This geometric score is the minimum curvature a collision-free backbone would need to reach the goal, estimated by a sampling-based shape planner. It depends only on the scene. Both scores are used only if they rank episodes similarly, with Spearman correlation above roughly 0.7. If they disagree, that disagreement is itself a finding: expert demonstrations would then be encoding stylistic choices rather than physical necessity.
Research Program
The work has four parts.
Scoring ManiSoft. We compute $D$ for all 6,300 ManiSoft scene-trajectory pairs.
Re-evaluation by quartile. We re-evaluate Diffusion Policy, OpenVLA, and RDT per quartile of $D$, using ManiSoft's released evaluation protocol.
The straight-line reach baseline. We build a scripted policy that moves the gripper along a straight Cartesian path to the grasp and place poses. It uses a simple inverse-kinematics routine and ignores obstacles. It is the soft-robot analogue of copy-source.
A controlled intervention in Elastica. Stratification on existing data is correlational, because high-demand scenes may also be cluttered and visually harder. To separate the causes, we construct matched scene pairs in Elastica, modelled on its structured-obstacle task (Case 3). The two scenes in a pair are visually and semantically near-identical. They differ only in obstacle placement, which forces a straight versus a curved solution. This lets us dial $D$ while holding visual clutter fixed.
A last, smaller component tests whether the measurement is actionable. We fine-tune Diffusion Policy with demonstrations reweighted toward the top quartile. We then check whether high-demand success rises without low-demand success falling. This is a probe of usefulness, not a proposed new method.
Baselines
Aggregate reporting (do-nothing reference). This is ManiSoft's own aggregate success rate. The strongest reported numbers are Diffusion Policy at 31.6% and OpenVLA at 30.4% in clean scenes, and OpenVLA at 27.0% versus Diffusion Policy at 2.2% in randomized scenes (ManiSoft). For RDT, the file gives no exact figure; no reported result found on Lacuna for RDT's overall ManiSoft success rate. We first reproduce the reported numbers with the same checkpoints or training recipe, the same seeds, and the same episode counts. Stratification is then applied to exactly those rollouts, so no extra tuning budget enters.
The strongest existing stratification. This is the breakdown ManiSoft already uses: clean versus randomized scenes plus object category. The Rubik's cube was easiest and shoes hardest. Our score beats this breakdown only under a specific condition. In a per-episode logistic regression predicting success, adding $D$ must significantly improve held-out log-likelihood over a model containing scene type, obstacle count, and object category.
The straight-line reach baseline. This scripted policy is the shortcut probe. It gets no training. Its rollouts use the identical scenes and seeds as the learned policies. The shortcut-regime claim holds if two things are true. First, its success on the lowest quartile must fall within the confidence interval of the best learned policy. Second, it must drop toward zero on the highest quartile.
Ablation isolating the physics. Two control stratifications keep the quartile machinery and replace $D$. The first uses random quartiles. The second uses quartiles of a purely visual complexity proxy, the pixel fraction occupied by clutter. If either reproduces the crossover pattern, then mechanical demand is not the operative ingredient.
Why not just…?
Why not just use the clean-versus-randomized split ManiSoft already reports? This is the real threat. Diffusion Policy's fall from 31.6% to 2.2% already shows masking. But the randomized condition changes lighting, textures, clutter, and obstacle geometry all at once. ManiSoft cannot say whether Diffusion Policy fails because it cannot see or because it cannot bend. The matched-pair Elastica experiment holds the visuals fixed and varies only required curvature, which answers that question directly. The regression test settles the correlational version. If scene type and obstacle count already explain failure and $D$ adds nothing, we will report that stratification by existing metadata is sufficient.
Why not just report worst-case success across random seeds or domain-randomization draws? A worst-case number tells you that a policy sometimes fails, not which kind of episode it fails on. The file's evidence is that randomized evaluation in ManiSoft did not isolate the cause of failure. The COIL-100 crossover was invisible without explicit strata (Aggregate Metrics Hide Shortcut Regimes). We will report worst-seed success alongside per-quartile results. The test is whether a policy ranking under worst-seed success differs from its ranking on the top quartile.
Why not just discover failure slices automatically from the policy's own errors? Slicing by where the policy fails is circular. It finds regions of low success by definition and cannot say whether those regions are hard for physical reasons. A physics-derived score fixed before any policy is run can be checked against scenes the policy has never seen. For comparison, we will still run a simple error-clustering method on rollout features. We then measure how much its discovered slices overlap with the high-$D$ quartile.
Why not just use expert effort, such as demonstration length or planning time, as the difficulty proxy? This is cheap and partly plausible. We include demonstration length as a covariate in the regression. If length predicts failure as well as $D$ does, the simpler proxy wins, and we will say so.
Proposed Experiments and Decisive Checks
The central experiment is per-quartile success for Diffusion Policy, OpenVLA, RDT, and the straight-line baseline across ManiSoft's four tasks: Collection, Alignment, Stacking, and Arrangement. Results are reported with bootstrap confidence intervals over episodes.
The proposal is supported if three things hold. First, the straight-line baseline must match learned policies on the lowest quartile. Second, all policies must fall sharply on the top quartile. Third, $D$ must add predictive power beyond existing scene metadata.
The proposal is refuted under either of two outcomes. One is that learned policies beat the scripted reach uniformly across quartiles, which would mean they genuinely exploit compliance. The other is that success is flat in $D$ once clutter is controlled.
The matched-pair Elastica study is the decisive causal check. If success falls as $D$ rises while visual scene statistics stay fixed, mechanical demand is a genuine and currently unreported axis of difficulty.
A secondary analysis addresses the "performance parity" claim of Bridging Embodiment Gaps. That paper finetuned VLAs on the Embuddy tendon-driven arm to 100% and 70% success on its first two tasks, matching a rigid UR5 arm. Its dozens of episodes per task could not be stratified. We recommend that such hardware studies report $D$ per trial, computed from shape estimates such as the factor-graph Cosserat estimator of Continuum Robot State Estimation with Actuation Uncertainty. We treat that as a protocol recommendation rather than an experiment we can run here.
The per-episode $D$ scores and quartile splits will be released so that future ManiSoft results can be reported by stratum.
References
- ManiSoft: Towards Vision-Language Manipulation for Soft Continuum Robotics
- Aggregate Metrics Hide Shortcut Regimes: A Complexity-Stratified Benchmark for Novel View Synthesis
- Rethink reporting of evaluation results in AI
- Coarse race data conceals disparities in clinical risk score performance
- HERB: Measuring Hierarchical Regional Bias in Pre-trained Language Models
- Elastica: A Compliant Mechanics Environment for Soft Robotic Control
- Bridging Embodiment Gaps: Deploying Vision-Language-Action Models on Soft Robots
- Continuum Robot State Estimation with Actuation Uncertainty