A mobile robot that drives to a kitchen drawer often fails to open it. The navigation system counts "within a meter" as arrival, but the arm's learned grasping skill only works when the base stops within a few centimeters of the spots it saw during training. Fleets of such robots in private homes could pool their experience to fix this, using federated learning, where each robot trains on its own data and shares only model updates, never camera footage. Recent evidence, however, says the obvious way of doing this fails. When manipulation policies are averaged across robots, they succeed at coarse tasks and score exactly zero on precise ones.
We propose to split the problem along its precision boundary. Each robot keeps its fine-grained manipulation policy strictly local and never averages it. What the fleet shares is a model of manipulation readiness: given a local 3D view around the target, the robot's arm geometry, and a candidate base pose, it predicts the probability that the robot's own policy will succeed from there. Readiness is a smooth, low-frequency quantity governed by shared physics such as reach, occlusion, and joint limits, which is the kind of function that averaging handles well. A small per-robot correction head absorbs the parts that are idiosyncratic, namely each policy's preferred viewpoints and each unit's calibration offsets. Precision then comes from moving the base, the error-tolerant variable, into the comfort zone of a policy that was never touched by averaging.
Core Claim
Two separately documented failures have a common cause and a common fix. The navigation-manipulation gap arises because manipulation policies collapse when the base parks at viewpoints outside their training distribution, as Mobi-π shows. Federated manipulation collapses because averaging erases the sharp, contact-level structure of precise policies, as FLAME shows with 0% success on square-peg insertion and spatula scooping under FedAvg, FedAvgM, FedOpt, and Krum. Our claim is that fleet-level sharing should target base placement conditioned on the local policy, not the policy itself. Concretely, we expect a federated readiness model with local correction heads to match per-scene optimization (Mobi-π) in manipulation success while needing far fewer on-site trials and no 3D scene reconstruction. We also expect it to beat a centrally trained, policy-agnostic affordance predictor, because "where can the arm reach" is not the same question as "where does this policy work."
How This Sits Relative to Prior Work
Three papers motivate the proposal by locating the gap. MoMa-Kitchen trains NavAff, a centralized predictor that maps object masks into point clouds and fuses them with arm-reach and base-height features to score floor positions. Its labels describe generic reachability, not the success of a particular deployed policy, and it notes that scalar morphology descriptors are insufficient. Mobi-π does condition on the policy. It scores base poses by how closely rendered views resemble demonstration frames, checks visibility, and searches with Bayesian optimization over a Gaussian-splat reconstruction of each scene. That approach requires a per-scene 3D reconstruction and demonstration frames at deployment time, and it learns nothing across robots. Aim My Robot reaches 3 cm / 1° median precision to objects but targets a fixed goal image rather than a policy-dependent sweet spot, and it is trained centrally.
Three federated papers constrain the design. FLAME rules out averaging precise policies, and its finding that offline action error does not predict closed-loop success pushes us toward training on binary rollout outcomes. FedVLN shares only the language encoder of a navigation agent and keeps visual and action layers local, reaching 67.6% versus 64.6% centralized on unseen R2R environments. It supplies the partial-sharing principle we reuse, but it stops at room-level navigation. Fed-EC clusters outdoor robots by visual embeddings and averages within clusters, raising the share of robots that benefit from federation from 60% to 100%. We borrow its participation metric and test its clustering as an alternative to local heads.
The closest idea one would write at this intersection is to federate the affordance module and keep the contact policy local. Our proposal differs from that in what gets shared: a policy-conditioned success predictor trained on each robot's own outcomes, not NavAff-style geometric affordance.
Research Program
Each client is a robot–home pair. It holds a local manipulation policy (trained however it was, and frozen), its arm morphology, and its own scenes. The shared model $g_\theta$ takes a point cloud cropped around the target, a morphology vector (a sampled reachability envelope rather than scalars, addressing the MoMa-Kitchen limitation), and a candidate base pose $p$. It outputs a readiness logit. A lightweight local head $h_{\phi_k}$ adjusts this logit using features only robot $k$ has: a density score of the current view against its own demonstration features, in the spirit of Mobi-π's in-distribution term, computed and kept on-device. Each client collects rollouts, recording the base pose tried and whether its policy succeeded. It then trains
$$\min_{\theta,\phi_k}\ \sum_{(o,p,y)\in\mathcal{D}_k} \mathrm{BCE}\big(\sigma(g_\theta(o,m_k,p) + h_{\phi_k}(o,p)),\ y\big),$$
where $o$ is the local observation, $m_k$ the morphology vector, $y\in\{0,1\}$ the rollout outcome, and $\mathcal{D}_k$ the client's private rollout log. Only $\theta$ is averaged. At deployment, the robot ranks candidate poses with the combined score, drives to the best one, and tries the next if the attempt fails. Because every failed attempt yields a label, the fleet improves continuously.
On the analysis side, we will directly test the mechanism behind the claim. For each client pair, we will measure the cosine agreement of client updates for the readiness model and for a FLAME-style manipulation policy. We predict high agreement for readiness and low agreement for the policy.
Baselines
The naive reference point is navigation to the object's centroid followed by the local policy, the "BC w/ Nav" condition in Mobi-π. We run it with the same local policies and the same scenes. Any method must clearly beat it for the problem to be real.
The strongest per-scene competitor is Mobi-π itself, run on each client with its splat reconstruction and Bayesian optimization, using the same number of on-site rollouts as our method's trial budget. The file reports only relative gains for Mobi-π (a 7.65× improvement over its best baseline in simulation and 2.38× in the real world), with no reported absolute simulation success rate found on Lacuna. We will therefore re-run it. Beating it means equal or higher success with fewer trials and without scene reconstruction, or higher success at a fixed small trial budget.
The strongest learned placement predictor is MoMa-Kitchen's NavAff, which reports Top-1 manipulation success of 72% and RMSE 0.147 on MoMa-Kitchen. We train it centrally on the pooled data, which is a privacy-violating upper-bound setting, with the same backbone size and tuning budget as ours. Beating it requires higher manipulation success with the clients' own policies, not higher affordance-map correlation.
For full-policy federation, we run FedAvg on an end-to-end navigation-plus-manipulation policy, plus a personalized variant that keeps the action head local, FedPer-style. FLAME predicts near-zero precision for the former. The latter is the serious federated competitor.
Ablations isolate the key ingredients. We remove the local head (shared model only). We train readiness on reachability labels instead of policy outcomes. We run local-only readiness with no federation. We replace local heads with Fed-EC clustering. Finally, we train our model centrally as a ceiling. The gap between local-only and federated readiness is the value of the fleet. The gap between outcome labels and reachability labels is the value of conditioning on the policy.
Why not just…?
Why not just run Mobi-π on every robot? This is the real threat, and we treat it as the main baseline. Mobi-π needs a reconstruction of each scene and demonstration frames at test time, and it starts from scratch in every new home. Its static-scene assumption also breaks when furniture moves. Our bet is that a fleet prior makes the first attempt good in a new home. If Mobi-π with an equal trial budget matches us on success and cost, the federated component adds nothing, and the experiments comparing trial budget against success will show it. In that case we would report the fleet prior as a warm start for Mobi-π's optimization rather than a replacement.
Why not just train NavAff centrally on simulation and ship it? Centralized geometric affordance ignores which viewpoints a particular policy tolerates. That tolerance is the whole failure mode Mobi-π identifies. RoboCasa also reports that real-world co-training was needed to go from 13.6% to 24.4% success, which suggests that on-site data matters. The ablation that swaps policy outcomes for reachability labels, plus the centralized-NavAff baseline, settles whether policy conditioning pays off.
Why not just federate the whole policy with a personalized local head, or just use visual servoing for the last centimeters? Personalization still averages the visual encoder that FLAME's precision tasks depend on, and we test it directly. Classical servoing cannot recover from a base parked where the target is unreachable or occluded, which is the reachability failure documented by Qiu et al. and MoMa-Kitchen. We include a servoing-after-centroid condition inside the naive baseline to quantify this.
Proposed Experiments
The primary testbed is RoboCasa, Mobi-π's platform. We build clients by crossing floorplans with MoMa-Kitchen's six arm morphologies and with local policies trained from deliberately different demonstration viewpoints, so that sweet spots genuinely differ across clients. A second testbed uses MoMa-Kitchen scenes to report NavAff's metrics alongside manipulation success. A third, decisive check adapts FLAME's square-peg and spatula tasks by making the arm's base offset a decision variable. Each client keeps an unaveraged local policy and only placement is federated. Non-zero success where FLAME's averaged policies scored 0% would be the cleanest evidence for the claim.
Metrics are manipulation success, attempts to first success in unseen homes, final base-pose error, the fraction of clients doing better than local-only training (Fed-EC's participation rate), and bytes uploaded as a share of total parameters, compared against FedVLN's 24.6%. We also report the update-agreement analysis described in the Research Program.
The proposal is falsified if federated readiness fails to beat local-only readiness on unseen homes, or if Mobi-π at an equal trial budget matches it. A small real-robot study in two mock kitchens with different arms would check that outcome labels collected on hardware carry calibration drift into the local heads as intended.
References
- Mobi-$\pi$: Mobilizing Your Robot Learning Policy
- FLAME: A Federated Learning Benchmark for Robotic Manipulation
- MoMa-Kitchen: A 100K+ Benchmark for Affordance-Grounded Last-Mile Navigation in Mobile Manipulation
- Aim My Robot: Precision Local Navigation to Any Object
- FedVLN: Privacy-Preserving Federated Vision-and-Language Navigation
- Fed-EC: Bandwidth-Efficient Clustering-Based Federated Learning for Autonomous Visual Robot Navigation
- RoboCasa: Large-Scale Simulation of Household Tasks for Generalist Robots
- ORB: Operating Room Bot, Automating Operating Room Logistics through Mobile Manipulation