Tuning Fuzzy Gain Schedulers on the Errors of Spatial Vision-Language Models

Robots that take instructions in language increasingly rely on vision-language models to decide where things should go: "put the plug in the socket left of the lamp." These models are poor at metric geometry. A two-pixel slip in the image can become a 5–10 cm miss on the table. Yet the low-level controllers that execute the motion use fixed stiffness and gains, so they cannot tell a confident target from a guessed one. We propose to make the low-level controller aware of how wrong the perception is likely to be. The controller is a fuzzy PID, a classical feedback loop whose three gains are adjusted online by a small table of human-readable if–then rules. Our version reads not only the tracking error and contact force but also the spatial model's own uncertainty about the target. When the target is uncertain it approaches softly and searches. When the target is confident it moves fast and stiff.

The second ingredient is how the rules get tuned. Fuzzy controllers in the literature are tuned on idealized simulations without friction or noise. We tune them with domain randomization, which means training across many simulated worlds with randomly perturbed physics. We perturb friction, mass, latency, and, unusually, the target error itself, using an error distribution measured from the actual spatial model. Tuning uses an episodic black-box optimizer rather than off-policy reinforcement learning. That choice lets us keep the integral term, which removes steady-state offsets and which reinforcement-learned fuzzy controllers have had to drop. In two sentences: the controller's gains are scheduled on the perception model's uncertainty, and the schedule is tuned across randomized physics and randomized perception error. The test is whether this beats a single robust fixed-gain compliant controller tuned the same way, and when.

Core Claim

The claim has a specific, falsifiable shape. Gain scheduling on perceptual uncertainty helps only if the spatial model's error varies from scene to scene, and only if a cheap signal predicts that variation. If the error is roughly the same size everywhere, one well-tuned fixed impedance (a fixed spring-damper relation between position error and force) should do as well, and the fuzzy machinery would be unnecessary. So the proposal is not "fuzzy beats PID." That comparison has been run many times, on simulations far from our setting. On a 2-DOF helicopter, Implementation of Fuzzy-PID Controller for 2-DOF Helicopter reports 1 s rise time and 6% overshoot against 7.5 s and 40% for linear PID. In simulated pneumatic grinding, Research on Fuzzy PID Compliant Grinding Control Based on DBO Algorithm Optimization reports 0.5 s stabilization and 0% overshoot against 3.0 s and 20%. The proposal is that the best gains for a contact approach depend on how far off the commanded target is, that a spatial model's uncertainty partly reveals this, and that a controller tuned across randomized physics and randomized perception error can exploit it on real hardware without fine-tuning.

How This Differs From Existing Work

Three lines of work motivate the gap.

Spatial vision-language models ground spatial language but stop at a target point. RoboSpatial builds 3 million spatial annotations and raises Kinova Jaco placement success from 23.7% to 52.6%. It executes open-loop to a predicted coordinate and itself flags the 2-pixel-to-5–10 cm amplification. SpatialThinker uses dense spatial rewards to reach 76.3% on RoboSpatial-Home. It outputs boxes and answers, never motions.

Sim-to-real manipulation handles contact either blindly or stiffly. Learning Insertion Primitives reaches 96.7% on square pegs and 86.7% on unseen Ethernet connectors with force-terminated primitives. It searches without vision and with fixed low-level tracking, and names vision-based hole localization as future work. ManiSkill2 uses fixed PD or velocity control and leaves 0.5 mm-clearance plug insertion unsolved.

Learned fuzzy controllers have never left low-dimensional simulators. Soft Fuzzy Reinforcement Neural Network PD learns gains with Soft Actor-Critic on Pendulum-v1, reducing integral absolute error by 40.19% versus plain Soft Actor-Critic. It drops the integral gain because a running error sum breaks the Markov assumption of its replay buffer. Digital Twin-Empowered Robotic Arm Control couples PPO with fuzzy PID but without randomized transfer or perception.

Two results constrain the design. RoboTwin 2.0 found that a vision-language observer judging execution failures reached only 0.431 accuracy and could not see force-level problems. The vision model therefore should not supervise contact; force feedback must. The file also notes that heavy visual randomization strips fine spatial detail. Based on that note, we do not ask the visual pipeline for millimetre precision. We ask it only for a target and an honest error bar.

We borrow method from two places. Per-episode dynamics randomization comes from Sim-to-Real: Learning Agile Locomotion For Quadruped Robots. Metaheuristic tuning of membership functions comes from the DBO paper and Self-Tuning Control of Manipulator Positioning Based on Fuzzy PID and PSO Algorithm. Neither has been tuned under randomized dynamics, and neither has been given a perception-uncertainty input.

Research Program

The system has three tiers.

Perception tier. A frozen spatial model, SpatialThinker-7B or the RoboSpatial-tuned LLaVA-NeXT, reads an RGB-D image and an instruction and proposes a 3D target. We obtain an uncertainty signal $\sigma$ without retraining. We sample the model $k$ times at nonzero temperature, lift each predicted point or box centre to 3D using depth, and take the spread of those $k$ points.

Error model. We fit $p(\epsilon \mid \sigma)$, the distribution of 3D target error $\epsilon$ given the spread $\sigma$. The data come from several thousand ManiSkill2 and RoboTwin 2.0 scenes rendered with visual randomization, where ground truth is known. The first scientific question is answered here, before any control work. If $\sigma$ does not predict $\epsilon$, the scheduling hypothesis is dead and we report that.

Controller. A Cartesian fuzzy PID with anti-windup runs at the arm's control rate. Its inputs are position error, its rate, sensed contact force, and $\sigma$. Its outputs are gain multipliers on $K_p$, $K_i$, $K_d$, plus a search-amplitude parameter for a spiral search once contact is made. The integral term is gated on contact so it cannot wind up during free motion. Membership-function breakpoints and output scalings form a parameter vector $\theta$ of roughly 40–80 numbers.

We tune $\theta$ by maximizing expected episodic return under joint randomization:

$$\theta^* = \arg\max_\theta \; \mathbb{E}_{\xi \sim P_{\text{dyn}},\; (\sigma,\epsilon) \sim P_{\text{perc}}} \big[ R(\theta; \xi, \sigma, \epsilon) \big]$$

Here $\xi$ collects randomized friction, link masses, actuator latency, and force-sensor noise. The pair $(\sigma,\epsilon)$ is drawn from the fitted perception error model. The return $R$ rewards insertion or placement success and penalizes time, peak contact force, and residual offset. We use CMA-ES and, for comparability with the fuzzy literature, PSO. Both score whole episodes, so the integral state never needs to be Markov. This is the precise reason we can keep $K_i$ where the Soft Actor-Critic fuzzy controller could not.

Inspection. The tuned rule surface is small enough to plot. We check whether it learned the intended behaviour: soft and searching under high $\sigma$, stiff and fast under low $\sigma$. A surface that ignores $\sigma$ would itself be a negative result.

Tasks and hardware. In simulation we use ManiSkill2 PickCube, its 0.5 mm-clearance plug insertion task, and peg-in-hole variants matching the insertion-primitives paper. Real transfer uses a 6-DOF or 7-DOF arm with a wrist force-torque sensor and an RGB-D camera. The real tasks are peg and connector insertion and language-specified placement in the RoboSpatial style, run zero-shot.

Baselines

The naive reference is the spatial model's target executed open-loop with the arm's stock fixed-gain position controller. This is effectively RoboSpatial's execution mode, and RoboSpatial reports 52.6% placement success on a Kinova Jaco. That is a different robot and task, so we rerun it on our setup rather than borrow the number.

The strongest simple competitor is a fixed-gain impedance controller plus the same contact-triggered spiral search. Its handful of gains are tuned with the identical CMA-ES budget under the identical randomization, including perception error. This is the baseline most likely to tie us, because randomization alone may find one compromise stiffness good enough everywhere. Beating it means higher success at matched or lower peak force, or matched success in less time, with the gap concentrated in the high-$\sigma$ scenes.

The strongest existing contact method is the force-terminated hybrid primitives policy of Learning Insertion Primitives. The paper reports 96.7% success on square pegs, 100% on waterproof connectors, and 86.7% on unseen round pegs and Ethernet connectors, all on a FANUC LR Mate. That policy received a much smaller initial position error than a vision model produces. We therefore run it from the same vision-proposed targets, retrained in our simulator for the same number of episodes.

For the high-precision simulated task, no reported result was found on Lacuna for ManiSkill2's 0.5 mm plug insertion beyond the authors' statement that it remains a significant challenge. Our sim-to-real reference there is ManiSkill2's PickCube at 91% in simulation and 60% on the real robot.

A learned alternative is residual Soft Actor-Critic that outputs gain multipliers from the same inputs. It is the end-to-end analogue of the fuzzy scheduler and gets the same simulator steps.

Three ablations each remove one ingredient while keeping the tuning budget fixed:

  • the fuzzy scheduler with the $\sigma$ input removed, which isolates the perception-uncertainty idea;
  • the full scheduler tuned on nominal physics without randomization, which reproduces the fuzzy literature's practice;
  • the full scheduler with $K_i$ forced to zero, which tests whether keeping the integral term matters under unmodeled loads.

Why not just…?

Why not just tune one robust impedance controller under domain randomization? This is the real threat, and we do not argue it away. The existing fuzzy-versus-PID numbers come from idealized single-plant simulations, and nothing in the file shows fuzzy scheduling beating a randomization-tuned fixed impedance under perception error. Our experiments settle it in two ways. First, the matched-budget comparison is stratified by $\sigma$. Second, a control condition draws perception error at a constant size regardless of $\sigma$. There, we predict the fixed controller ties the fuzzy one. If the fixed controller also ties under realistic, varying error, we report that scheduling is unnecessary.

Why not just compose RoboSpatial with the insertion primitives and stop? The primitives already handle contact robustly. But they were validated from small initial errors, and their fixed search pattern cannot know whether the hole is 2 mm or 5 cm away. The composition is one of our baselines. Our prediction is specific: it should lose time and success mainly when the vision error exceeds the primitive's search radius. If it does not, then a better perception model, not a smarter controller, is the right investment.

Why not just learn the gain policy with deep reinforcement learning instead of fuzzy rules? On Pendulum-v1, Soft Fuzzy Reinforcement Neural Network PD found that adding fuzzy structure to Soft Actor-Critic reduced integral absolute error by 40.19% and steadied training compared with Soft Actor-Critic alone. That evidence is from a toy system, so our residual Soft Actor-Critic baseline tests the question directly. The fuzzy scheduler also offers two concrete advantages that the comparison will price: an inspectable rule surface, and cheap evaluation at kilohertz rates.

Why not just fix perception, for example by fine-tuning the spatial model further? RoboSpatial fine-tuning roughly doubled accuracy, from 30.3% to 60.5%, yet still leaves centimetre-scale errors after projection. Separately, Vision Language Models See What You Want but not What You See found that scaling did not improve perspective-taking. Better perception narrows the error distribution but will not remove it. We test this by swapping in the stronger spatial model. The scheduler's advantage should shrink as error shrinks, and the size of that shrinkage tells practitioners where to spend effort.

Proposed Experiments

The first decisive check runs before any control work: does sampling spread predict 3D error? We measure the rank correlation between $\sigma$ and $|\epsilon|$ on held-out simulated scenes and on RoboSpatial-Home images that have depth.

The second check is the main simulation study. All baselines and ablations are tuned with the same episode budget, then evaluated on unseen randomization seeds and unseen object geometries. We report success rate, time to completion, peak contact force, and residual steady-state offset under added constant payloads. The offset metric is the one designed to expose the zero-$K_i$ ablation.

The third check is zero-shot real-robot transfer on peg, connector, and language-specified placement tasks. We run at least 30 trials per condition so that differences of the size seen in the insertion-primitives results are distinguishable from noise. Results are reported per $\sigma$ tercile.

The proposal succeeds if three things hold. The $\sigma$-ablated scheduler and the fixed impedance controller fall behind specifically in the high-uncertainty tercile. They tie in the constant-error control condition. And the nominal-physics-tuned scheduler degrades on hardware, as the friction caveat in the DBO paper predicts. Any other pattern still answers the question of whether perception-aware compliance is worth building, and we report it as such.