When a household robot tidies a home alongside a person, the expensive mistakes are social. It walks to the cup the person is already reaching for. It waits for a drawer the person cannot open from where they stand. It retries a failed grasp when the object actually failed to appear because the partner moved it. The PARTNR benchmark measures this directly. Language-model planners paired with partners show a 300% increase in wasted "extraneous effort", take 1.5x more steps than human pairs, and fall from about 73% success with perfect state to 30% under realistic perception (PARTNR). One kind of method guesses what other agents intend, called latent strategy inference. The best example predicts a partner's upcoming subgoal, meaning the state it is heading toward, rather than its next motor command (Opponent Modeling based on Subgoal Inference). That work runs only in gridworlds and StarCraft, where any goal can be reached. In a real house, what a partner can do next is tightly limited: the objects within reach, the doors that must be opened first, and the path length of its body.
This note proposes using that limit as information. A robot that already has value functions scoring how feasible its own skills are can build the same kind of scores for its partner's body. It can then use them as a prior over what the partner is about to do. The resulting belief over partner subgoals is grounded in physical feasibility. It feeds a two-level controller. A high-level planner picks the robot's next subgoal at the moments when a skill finishes, and frozen low-level skills carry it out. Our claim is that this structure lets the robot anticipate its partner earlier and from less data. It should also tell "my grasp failed because physics" apart from "my grasp failed because my partner took the object", which current planners cannot do.
Core Claim
The closest existing pieces each cover half of the problem. SayCan scores each skill by an LLM's task relevance times a learned value function's physical feasibility ("Can"). This gives 84% planning and 74% execution success on real kitchen tasks, but only for a single robot (Do As I Can, Not As I Say). OMG infers partner subgoals with a conditional variational autoencoder over a fixed horizon $H$. It reaches roughly 70–90% win rate against unseen SMAC opponents, where action-prediction baselines reach about 50–60% (OMG). However, its subgoals are unconstrained abstract states, and its fixed horizon is a known source of fragility. Our contribution is to turn SayCan's feasibility term into a model of the partner, and to combine it with an amortized learned prior in the style of OMG. We maintain a belief over the partner's current subgoal $g$, drawn from a discrete set of (object, receptacle) pairs:
$$b_t(g) \;\propto\; q_\phi(g \mid h_\tau)\;\prod_{k=\tau}^{t-1} \exp\!\big(\beta\,[V_p(s_{k+1}, g) - V_p(s_k, g)]\big).$$
In this equation, $h_\tau$ is the interaction history up to the partner's most recent skill boundary $\tau$. $q_\phi$ is a learned encoder trained on a population of partners. $V_p(s, g)$ is a value function estimating how close the partner's body is to being able to achieve $g$ from state $s$. $\beta$ is a rationality temperature. The product term says the partner is more likely pursuing goals it is visibly making feasible progress toward. Two design choices separate this from OMG. First, subgoal segments begin at observed skill boundaries such as a pick or place completing, not at a fixed horizon $H$, which removes the horizon hyperparameter. Second, goals that the partner's body cannot reach get near-zero mass no matter what the learned encoder says.
The testable claim has three parts. Compared with a learned encoder alone, the grounded belief should identify the partner's subgoal earlier within the partner's skill execution. It should need fewer partner-interaction samples to train. It should generalize better to unseen houses and partner styles. These advantages should translate into lower extraneous effort and fewer steps in team rearrangement.
Research Program
The system is built in Habitat 3.0, which simulates articulated humanoid avatars alongside a Spot-style robot at up to 1,191 FPS (Habitat 3.0). We use its social rearrangement task, and then the PARTNR task suite on the same simulator.
The first component is a pair of affordance value functions, one per embodiment. For the robot, we train feasibility values for the existing navigate, pick, place and open skills, as SayCan does. For the humanoid, we train the same values by rolling out Habitat's humanoid skill controllers. In those rollouts the humanoid's own reachability, path length and articulated-object access determine success. Training these values does not require any partner-modeling data, which is where much of the sample saving should come from.
The second component is the learned encoder $q_\phi$. It is trained on trajectories from a population of scripted and learned partners, in the style of Habitat 3.0's population-based evaluation. Its target is the partner's next skill-boundary subgoal. Its architecture follows OMG's conditional autoencoder.
The third component is the robot's high-level planner. It picks subgoals at the robot's skill boundaries, conditioned on the full belief $b_t$ rather than a single guess. It is trained by reinforcement learning with low-level skills frozen. This follows the offline-then-online recipe of GLIDER, which freezes low-level skills and adapts only the high-level policy (Divide and Conquer). The difference is that GLIDER has no other agents.
The fourth component is failure attribution. When a robot skill fails, the planner computes the posterior probability that the partner was pursuing a conflicting subgoal. If that probability is high, the failure is labeled social and the planner replans away from the object. If it is low, the failure is labeled environmental and the planner retries or recovers. PARTNR identifies this missing distinction as the reason planners loop after errors.
Using a full belief rather than a point estimate matters when anticipating the partner. Early in a partner's walk, several goals are feasible and remain plausible. The right robot action is often one that is useful under all of them, and a planner conditioned on the most likely goal cannot represent that hedge.
Baselines
The naive reference point is the robot's hierarchical planner with no partner model at all. It sees the partner only through ordinary observations. This is Habitat 3.0's Learn-Single-style agent. It tells us how much any partner reasoning is worth.
The strongest existing learned competitor on this exact task is the population-trained planner of Habitat 3.0. Its Plan-Pop3 and Plan-Pop4 baselines reach ZSC-pop-eval success around 71.7% on social rearrangement (Habitat 3.0). We rerun it with the same skills, the same partner population, and the same environment-step budget as our method. Beating it means higher zero-shot success with unseen partners, at equal or lower steps.
On PARTNR, the strongest reported reference is the LLM-based hierarchical planner: about 73% success with privileged state and 30% with realistic perception (PARTNR). We compare our method with that planner, giving the LLM the partner's recent actions in its prompt. We also test a variant in which our belief $b_t$ is serialized into the LLM's prompt. This tells us whether the belief helps even without learned high-level control.
The key ablation ports OMG directly to this setting. It uses the learned encoder $q_\phi$ alone, with fixed horizon $H$ swept over several values, and no feasibility term. We tune it with the same budget as our method. Beating this ablation is the core result, because it isolates physical grounding as the active ingredient. We found no reported result on Lacuna for OMG in Habitat or any 3D embodied benchmark.
Further ablations isolate the remaining ingredients. One drops the learned prior and keeps only the feasibility-based inverse-planning term. One replaces skill-boundary segmentation with a fixed horizon. One replaces the full belief with its most likely goal. One swaps the humanoid's own affordance values for the robot's values applied to the humanoid's state, which tests whether modeling the partner's body specifically matters.
Why not just…?
Why not just use a proximity heuristic that marks objects the partner is near or carrying as claimed? This is the most serious threat, because many conflicts in tidying become obvious once the partner is close. The heuristic can only react, though. It fires late in the partner's approach, after the robot may have already committed to a long walk. Our claim is specifically about anticipation lead time. We will measure subgoal-identification accuracy as a function of the fraction of the partner's skill already completed, and include the heuristic as a baseline. If the heuristic matches our method on success and extraneous effort, the anticipation claim is wrong for household-scale tasks. We would report that result and restrict the method to the failure-attribution component.
Why not just train OMG's encoder on enough partner data, letting it learn feasibility implicitly? With unlimited in-distribution data this may well work, so the experiments compare data-scaling curves rather than single points. The grounding term should matter most when partner data is scarce, and in unseen houses where feasibility comes from new geometry the encoder has never seen. If the two curves meet early and stay together in unseen houses, the prior adds nothing and the idea fails.
Why not just let LLM agents talk, as in RoCo, or evolve plans from failures, as in EvoCF? RoCo coordinates robots through multiple rounds of LLM dialogue checked by motion planners. Its authors acknowledge this is high-latency and open-loop (RoCo). EvoCF induces rules from failures and reaches 0.84 success on MAP-THOR, but slows down with congestion at four agents (EvoCF). Neither applies to a human partner who will not join a dialogue. The PARTNR LLM-with-history baseline above is our direct test of the prompting route. The belief-in-prompt variant tests whether the two routes are complementary.
Why not just train flat multi-agent reinforcement learning at high simulator throughput? COMRAD ran seven flat algorithm families for 100M steps at about 25,000 FPS. Its grounded-signaling task stayed completely unsolved (COMRAD). We therefore include a flat population-trained policy only as a sample-efficiency reference point, not as a serious competitor.
Proposed Experiments
The first experiment measures anticipation, before any control is involved. On held-out Habitat 3.0 houses and held-out partner styles, we measure top-1 subgoal accuracy and calibration (expected calibration error) of $b_t$ at 10%, 25% and 50% of each partner skill segment. We compare our method, the OMG port, feasibility-only inference, and the proximity heuristic. We then repeat the measurement with partner training data cut to 10% and 1%. The decisive check is whether the grounded belief keeps its accuracy at the low data budgets while the learned encoder alone degrades.
The second experiment measures team performance on Habitat 3.0 social rearrangement against the Plan-Pop baselines. We report zero-shot success with unseen partners, steps to completion, and environment steps needed for the high-level planner to reach 90% of its final success. The last metric is our sample-efficiency measure.
The third experiment moves to PARTNR. We report success, percent complete, steps, and extraneous effort under both privileged and realistic perception. The main comparison is the 300% extraneous-effort gap and the 30% realistic-perception success reported for current planners. Realistic perception corrupts both the history and the feasibility estimates. This tells us whether physical grounding degrades gracefully with noisy perception or makes the collapse worse.
The fourth experiment tests failure attribution. We inject controlled failures into robot skills: physical ones, such as dropping the object or a blocked grasp, and social ones, such as a scripted partner taking the target first. We score attribution accuracy and the resulting reduction in replanning loops.
The proposal is supported if grounded inference beats the OMG port on unseen houses at low data budgets, and that advantage shows up as lower extraneous effort on PARTNR. It is refuted if the proximity heuristic or a data-rich learned encoder closes the gap.
References
- PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks
- Opponent Modeling based on Subgoal Inference
- Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
- Habitat 3.0: A Co-Habitat for Humans, Avatars, and Robots
- Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
- RoCo: Dialectic Multi-Robot Collaboration with Large Language Models
- EvoCF: Multi-Agent Collaboration via Agentic Memory-Driven Evolutionary Counterfactual Planning
- COMRAD: A Benchmark for Embodied Cooperative Multi-Agent Reinforcement Learning