Touch as a Shared Anchor for Federated Contact-Rich Manipulation

When many robots learn one skill together without pooling their data, each one sees a different world. The lighting differs, the camera sits at a different angle, the table has a different texture. Federated learning is training in which each robot updates a shared model on its own private data and sends only the weight changes to a server, which averages them. The largest simulated benchmark of this setting, FLAME, reports that every standard averaging rule reaches 0% success on peg insertion and spatula scooping, while the same rules succeed on coarse tasks such as closing a box. Its policies see only camera images and joint angles. Separately, work on the "haptic feedback gap", meaning the lack of force and touch information in robot demonstrations and policies, shows that vision-only policies fail at exactly these contact-heavy moments. They fail because the gripper hides the point of contact from the camera and because a camera cannot measure resistance.

Our claim joins these two findings. Force and touch signals barely change across the variations that separate federated clients, since lighting, camera pose and texture leave a contact wrench unchanged. A force pathway trained across a fleet should therefore disagree far less from client to client than a visual pathway does. We propose to exploit this asymmetry directly. The contact pathway is averaged across all robots as a shared anchor. The visual pathway is partly kept local. Each robot's force sensor is corrected for its own calibration by a small private layer. The prediction we test is specific: the gap between federated and centralized training should be much smaller for force-aware policies than for vision-only ones, and that gap should be concentrated in the visual parameters.

Core Claim and What Is New

We make two claims, and each can fail independently. The first is diagnostic. On FLAME's precision tasks, the collapse to 0% comes from two different causes: missing contact information, which would also sink a centralized policy, and visual client drift, where local models pull toward conflicting solutions. FLAME does not separate these causes, and the file reports no centralized result on its precision tasks. We measure both. The second claim concerns the method. When the model has separate pathways for vision and for force, the server should aggregate them differently. Contact features are shared widely. Visual features are handled per client or per cluster. Force inputs pass through a private per-client calibration layer that is never uploaded.

The closest prior work falls short of this in different ways. FLAME supplies the benchmark and the failure, but it has no contact modality and gives no explanation for the failure. Federated Imitation Learning federates heterogeneous sensors for driving by rendering every modality in a cloud simulator and taking the median of client predictions. That approach depends on the server being able to simulate each client's world. It cannot handle contact, whose mechanics cannot be rendered in advance. The survey Federated Learning for Large-Scale Cloud Robotic Manipulation names tactile bandwidth and sensor heterogeneity as open problems but offers no algorithm. Our design does not require a server-side simulator, and it treats modality, not client, as the unit of aggregation.

Research Program

The policy follows the decoupled design of DECO. DECO keeps vision and action in a main transformer and brings touch in through a separate cross-attention adapter. It fine-tunes that adapter with low-rank updates, 7.97M of 91M parameters, and reports that naive fusion lets tactile noise corrupt the visual features. We reuse this separation for a federated purpose DECO never considered: the small adapter becomes the part that is cheap to communicate and widely shared.

For action targets we adopt DexForce. DexForce relabels each demonstrated position $x_o$ as $x_f = x_o + k_f f$, where $f$ is the measured contact force and $k_f$ is a single stiffness constant. This pushes the command into the contact so that the policy actually exerts force; policies trained on raw positions "failed almost entirely."

Sensor calibration is where federation causes a new problem. If a client's sensor reports $\tilde f = g_i f + b_i + \epsilon$, with unknown gain $g_i$, offset $b_i$ and noise $\epsilon$, then each client's relabeled targets and force inputs are silently distorted in a different way. Each client therefore learns a private affine map (a per-channel scale and shift) on its force input. This map is trained locally and never sent to the server, in the spirit of keeping client-specific normalization local.

On the server, the force adapter and the action head are averaged over all clients. The visual encoder is aggregated within clusters of clients whose visual updates point in similar directions, or is fine-tuned locally. We choose between these two by validation.

Alongside the method we run an instrumented analysis. At every round we log the pairwise cosine disagreement between client updates, separately for each module. This tests the central mechanism directly: force-module updates should agree across visually perturbed clients much more than visual-module updates do.

Baselines

The naive reference point is FLAME's own vision-and-joint-angle policy trained with Federated Averaging (FedAvg, plain weight averaging), under FLAME's protocol of about 25 local epochs and its 400/20 environment split. FLAME reports 0% success on Insert Onto Square Peg and Scoop With Spatula for FedAvg, FedAvgM, FedOpt and Krum. It reports 84% (FedAvg) and 86% (Krum) on Close Box, and 28% (FedOpt) on Slide Block. We rerun it rather than copy these numbers, so that seeds and evaluation episodes match.

The same vision-only policy trained centrally on pooled data is essential. It tells us whether FLAME's zero is a federation failure or a sensing failure. No reported result was found on Lacuna for centralized training on FLAME's precision tasks.

The strongest existing competitor is the obvious combination: add force inputs and DexForce-relabeled targets to the policy and run FedAvg on everything. Its centralized analogue has strong published support. DECO reports 82.50% average success with its tactile adapter against 72.25% vision-only, with a 20-point gain on assembly and waste disposal. DexForce reports 76% average success across its six tasks. A centralized force-aware policy is the upper bound.

We add two cheaper fixes for calibration: per-client z-scoring of force channels, and FedAvg with robust aggregation (Krum).

The ablations isolate each ingredient. We remove split aggregation, so everything is averaged. We remove the private calibration layer. We remove DexForce relabeling and keep raw positions. And we fuse force by simple concatenation instead of the decoupled adapter.

To keep the comparison fair, every arm uses the same demonstrations, the same client partition, the same number of rounds and local epochs, and the same number of evaluation episodes. Each arm also gets the same hyperparameter search budget of equal trial count on the validation environments. The proposal beats the strongest baseline if it wins on the precision tasks by a margin outside seed-level confidence intervals, and if it closes most of the gap to the centralized force-aware upper bound without losing ground on Close Box.

Why not just…?

Why not just add force to the input and run FedAvg? This is the real threat. If force alone fixes the problem, our aggregation machinery is unnecessary. The experiments settle it directly. Without simulated calibration differences, we expect plain force-aware FedAvg to recover much of the precision gap, and we will report it honestly if it recovers all of it. Our claim rests on two conditions: under realistic per-client gain, offset and noise, plain averaging should degrade, and the private calibration layer and split aggregation should recover what is lost. If both of those fail to appear, the paper becomes a clean negative result with a diagnostic contribution.

Why not just per-client z-scoring of force? Z-scoring removes offset and scale in the input statistics. It cannot tell a weak sensor from a client whose tasks involve lighter contacts. It also does not correct the relabeled targets $x_f$, whose error depends on the true gain. We include it as a baseline and expect it to match the learned layer under pure gain and offset shifts but lose under task-dependent force distributions.

Why not just use an impedance controller on the vision policy? Autonomy Infused Teleoperation argues that compliance protects hardware but leaves the policy uninformed about contact. In the GELLO force study, the position-only policy pulled through empty air after a missed grasp, reaching 62% against 93% with force. We still run a compliant vision-only arm in simulation.

Why not just train centrally or in simulation? Pooled training is our upper bound, not our rival. The case for federation rests on privacy and bandwidth (survey) and on the tactile reality gap that Tabero concedes. We note one tension. GELLO reports force hurting nut assembly, 60% versus 42%. We therefore report per-task results rather than averages, and we check whether the decoupled adapter avoids that loss.

Proposed Experiments

Benchmark regeneration. We regenerate FLAME demonstrations for Insert Onto Square Peg, Scoop With Spatula, Slide Block and Close Box. During regeneration we record the wrist wrench and joint torques from the physics simulator behind RLBench. If the released data already contain them, we use them directly.

Calibration heterogeneity. We add simulated per-client sensor gain, offset and noise at three severity levels, using the contact-force and GelSight-style streams in Tabero's Isaac Lab pipeline as a reference for realistic noise.

Real tactile data. We federate DECO-50 with its four scenarios as non-identically distributed clients. We report offline action error and, where hardware allows, success rates on its assembly tasks.

Metrics. We report success rate, the federation gap (centralized minus federated success) for each modality, per-module update disagreement, uplink bytes per round, and peak contact force.

Decisive checks. The mechanism predicts two results. First, force-module updates should agree across clients more than visual-module updates do under purely visual perturbations, and this advantage should shrink when FLAME's physical-parameter perturbations dominate. If force updates disagree as much as visual ones, the core premise is false. Second, the federation gap on peg insertion should be substantially smaller for the proposed method than for vision-only FedAvg. If centralized vision-only training also scores near zero, then FLAME's collapse was a sensing failure all along, and the contribution becomes showing that touch, not a better averaging rule, is what federated manipulation lacked.