When a fleet of home robots learns a shared perception model, the expensive part is often not labeling or uploading data but driving to the place where a useful image can be taken. Every meter costs battery, time and wear. Federated learning keeps raw images on each robot and shares only model updates, and it already has ways to decide which clients or which stored samples matter most. Those methods assume the data is already on disk. Robotics has the opposite habit. Cost-aware active perception, meaning methods that pick where to move by weighing expected information against travel cost, judges information by what the individual robot does not yet know about its own surroundings. We propose to join the two. Each robot scores candidate viewpoints by how much an image from there is expected to help the shared fleet model, divides that by the meters needed to reach it, and drives accordingly. The score uses a tiny per-class summary aggregated across the fleet instead of any raw data.
The reason to expect a gain is a finding from federated active learning. LoGo showed that a client's own view of uncertainty and the global model's view pull in different directions when client data differs strongly across clients. In an embodied fleet this disagreement has a real price, because a detour cannot be undone. A robot that spends meters resolving a corner of its own map full of chairs the fleet has seen thousands of times wastes energy. A robot that skips a room holding an object category rare across the fleet loses data no other robot can supply. We claim that viewpoint value should be defined against the federated model and the fleet's coverage, and that doing so yields a better shared model per meter driven. We also predict that this advantage grows with how different the robots' buildings are.
Core Claim
Consider robot $i$ in a federated round. It holds the broadcast global segmentation model $g$, its own partial 3D semantic map $M_i$, and a fleet summary vector $s$ with one entry per object class $c$. The vector $s$ is built by secure aggregation and differential-privacy noise from per-class counts and per-class mean predictive entropy of $g$ on each robot's recent frames. For a candidate viewpoint $v$, the robot renders from $M_i$ the expected number of not-yet-observed pixels $\hat{n}_c(v)$ of each class. It scores
$$U_i(v) = q_i(v)\sum_{c} \hat{n}_c(v)\, \frac{h_c}{\sqrt{1 + m_c}},$$
where $h_c$ is the fleet-mean entropy of $g$ on class $c$, $m_c$ is the fleet-wide labeled count of class $c$ (both read from $s$), and $q_i(v)\in[0,1]$ is a predicted measurement-quality factor. The factor $q_i(v)$ comes from viewing distance, lighting, and pose-uncertainty estimates. The square-root damping is the usual count-based bonus form, under which a class's value falls as its estimate stabilizes.
The robot then sequences viewpoints to maximize total $U_i$ per meter traveled within a per-round travel budget. It solves this with the minimum-ratio travelling-salesman sequencing of Liu et al., which used it to maximize mapped area per unit distance. The images it collects are labeled, it trains locally, and it uploads only weights.
What is new is the value function. ActiveSGM scores viewpoints by local geometric coverage and local semantic-label entropy, and here it supplies the map and rendering machinery. Burgard et al. and Taniguchi et al. subtract motion cost from information gain for a single robot's own state estimate. None of these asks what the fleet needs.
Research Program
The work starts with a simulated fleet in Habitat using habitat-lab on HM3D Semantics scenes. HM3D Semantics has 142,646 annotated object instances and serves as a pixel-accurate labeling oracle for whatever views a robot chooses to capture. Each of 20 to 50 simulated robots is assigned a few buildings. We control how different the robots' data are by grouping scenes by type and by the class histogram of their objects, in analogy to the Dirichlet skew used in KAFAL and IFAL. The shared model is a compact semantic segmentation network sized for a phone-class processor. Aggregation uses FedAvg-style averaging. That is adequate for a feedforward segmenter; the permutation-symmetry failures documented by FLEET-MERGE were for recurrent policies, and we will check that naive averaging is not the bottleneck by also running a centrally trained model on the same collected frames.
The program has three parts. First, we build the scorer and the cost-ratio planner on top of an ActiveSGM-style Gaussian-splat semantic map. Second, we build the fleet summary channel: a vector of $2C$ numbers per round, aggregated securely, with the differential-privacy noise level as an experimental variable. This answers the selection-signal leakage concern raised for Active Federated Learning, which uploads a raw loss per client. Third, we build the quality factor $q_i(v)$. It is motivated by Napolitano et al., who found that chasing model uncertainty alone drives a robot into regions where its sensors are unreliable. In our setting this matters because blurry, dark or pose-drifted frames produce high entropy that the fleet summary would mistake for scarcity. Simulated degradations (motion blur, low light, injected pose drift of the kind OpenBot-Fleet reports for phone tracking) make this testable. A small real pilot on OpenBot-class robots is a stretch goal, not a dependency.
Baselines
The naive reference is nearest-frontier exploration with FedAvg. Each robot explores its buildings with the classic nearest-frontier rule until the per-round travel budget is spent, captures frames at a fixed interval, and trains. This is what one gets by bolting standard exploration onto standard federated learning. Liu et al. report their cost-aware sequencing improves exploration efficiency by 10 to 30 percent over such baselines for coverage. Beating it on model quality per meter is the minimum bar.
The upper reference is exhaustive coverage with central pooling. Robots sweep their buildings fully and all frames are pooled centrally, as OpenBot-Fleet does with its cloud buffer. This bounds achievable accuracy and shows how much travel the proposal saves. Li et al. set the pattern we hope to reproduce: their cost-sensitive agent matched exhaustive search at 97.2 percent accuracy using 11.8 percent of the movement steps.
The strongest embodied competitor is ActiveSGM-style local active mapping with FedAvg. Each robot scores viewpoints by its own map's coverage and semantic-label entropy, minus travel cost, using the same planner and travel budget as ours. On Replica, ActiveSGM reaches about 85 percent mean intersection-over-union and reaches high completeness in 777 steps against 2,000 for passive methods. It is the most capable cost-aware perceiver in the file, and it differs from ours only in whose uncertainty it measures. There is no reported result found on Lacuna for any method on federated semantic segmentation across HM3D Semantics clients, so we will establish all numbers ourselves.
The strongest federated competitor is passive collection followed by LoGo or KAFAL sample selection. Robots drive nearest-frontier, store every frame, and then apply LoGo or KAFAL to choose which frames to label under the same label budget. On static benchmarks LoGo beat BADGE 12.0 times for every 2.4 losses, and KAFAL beat BADGE by about 3 percent on CIFAR-10.
Fairness rules apply to every method. All methods get the same scenes, client partition, network, local epochs, number of rounds, label budget and per-round travel budget. Every method's travel-cost weight or label threshold is tuned with the same number of trials on validation buildings.
Three ablations isolate the key ingredient. The first replaces fleet scarcity $h_c/\sqrt{1+m_c}$ with the global model's per-pixel entropy, which keeps the global model but drops the fleet summary. The second removes the quality factor $q_i(v)$. The third removes the travel denominator. We will count the proposal as beating the field only if it meets one condition: at matched total fleet meters, it beats the local ActiveSGM-style baseline and the passive-plus-LoGo baseline on held-out-building mean intersection-over-union, and on 10th-percentile per-robot accuracy, the tail metric that LEAF argues matters most under heterogeneity. That gap must hold across three seeds and every heterogeneity setting above the lowest.
Why not just…?
Why not just run local cost-aware active mapping on each robot and average the models? This is the real threat. If robots' buildings are similar, local uncertainty and fleet need coincide, and the local baseline should match us. LoGo's result is evidence only on static images that the two diverge under heterogeneity. The heterogeneity sweep below settles the question. We predict the gap is near zero when buildings are similar and widens as they diverge. If it stays near zero throughout, the fleet channel is not worth building, and we will report that.
Why not just drive cheaply, store everything, and let federated active learning pick what to label? Post-hoc selection is strong when cost lies in labeling or upload. It cannot recover a view the robot never reached, and LoGo, KAFAL and IFAL all assume candidates already exist on disk. We test this by varying the travel budget relative to building size. When budgets are generous, post-hoc selection should catch up. When they are tight, only movement-aware scoring can reach rare-class rooms. Where the crossover falls is itself a finding.
Why not just pool all frames centrally? For accuracy, pooling is an upper bound, not a competitor, and we report it as such. The proposal's case rests on travel and on keeping raw home imagery on the robot. OpenBot-Fleet uploaded more than a million frames from 72 homes. KAFAL also reports that training on 35 percent of selected data beat training on all of it, which suggests careful selection need not cost accuracy.
Why not just use the global model's entropy without any fleet summary? This is the first ablation, and it may come close. Entropy alone cannot tell "rare and uncertain" from "noisy and uncertain," and it ignores how many labels of a class the fleet already holds. If the ablation matches the full method, we drop the summary channel and its privacy cost.
Proposed Experiments
The main experiment plots held-out mean intersection-over-union against total fleet meters for every method. It is run at four heterogeneity levels and three per-round travel budgets, with 20 robots over 30 rounds. Success means our curve lies above the local baseline's curve at every heterogeneity level except the lowest, with the gap growing as heterogeneity rises.
The second decisive check counts the fraction of fleet-rare classes that each method ever captures. It tests the mechanism directly: rare classes reached, and fewer meters spent on classes the fleet already knows well.
The third check injects sensor degradation in 30 percent of rooms. Without $q_i(v)$, the fleet scorer should be pulled toward degraded rooms and lose accuracy. With it, the scorer should not. This tests Napolitano et al.'s exploration-versus-reliability tension in a federated setting.
The fourth check sweeps the differential-privacy noise on the fleet summary to find the privacy level at which the advantage over the local baseline disappears.
Together these checks state concretely when the fleet's view should steer a robot's wheels, and when it adds nothing over each robot's own curiosity.
References
- Re-thinking Federated Active Learning based on Inter-class Diversity
- Enhancing autonomous exploration for robotics via real time map optimization and improved frontier costs
- Understanding while Exploring: Semantics-driven Active Mapping
- Active Mobile Robot Localization
- Active exploration based on information gain by particle filter for efficient spatial concept formation
- Habitat-Matterport 3D Semantics Dataset
- Knowledge-Aware Federated Active Learning with Non-IID Data
- Inconsistency-Based Federated Active Learning
- Robot Fleet Learning via Policy Merging
- Active Federated Learning
- Active Sensing for Data Quality Improvement in Model Learning
- OpenBot-Fleet: A System for Collective Learning with Real Robots
- Robotic Occlusion Reasoning for Efficient Object Existence Prediction
- LEAF: A Benchmark for Federated Settings