PAIR: Bringing Household Robots into the Smart Home

Junseong Pyo, Hyeongjin Kim, Jongwook Han, Seungwon Kook, Sungjib Lim, Yohan JoarXiv 2026 · arXiv

TL;DR

PAIR (Physical-API Integration Environment for Robots) is a novel simulation framework that allows household robots to interact with smart appliances using both physical manipulation (pressing buttons, turning knobs) and digital APIs grounded in the industry-standard Matter protocol. Despite the theoretical efficiency of using APIs, the authors find that current state-of-the-art robot policies fail to complete even a single task in this environment, primarily because they struggle with the prerequisite physical challenges like navigation and grasping.

What Problem Does This Paper Address?

In a modern smart home, a human might start a dishwasher by pressing a button or by using a phone app. However, existing Embodied AI (EAI) simulation environments treat appliances strictly as physical objects. If a robot needs to "wash the dishes," current benchmarks assume it must physically actuate every control. This ignores the reality of the "Smart Home," where many functions are accessible via Application Programming Interfaces (APIs).

The challenge is that a robot needs to decide how to act. Should it navigate across the room to check a timer on a microwave's physical display, or should it simply query the device's state via the local network? To study this, we need an environment where physical actions and API calls are "coupled"—meaning they both affect and reflect the same underlying device state.

Methods

The authors introduced PAIR, built on top of the OmniGibson physics engine. The core innovation is the API–Physical State Coupling.

Figure 2. Architecture of PAIR. Agents interact with the physical simulation through physical actions and with a Matter-based API server through structured API calls. Both channels are coupled through a shared device state.

Key components include:

  • Matter Protocol Integration: Device states, commands, and constraints are modeled after the Matter protocol, the current industry standard for IoT interoperability.
  • Functional Assets: The team developed 84 smart-device assets (across 14 appliance and 4 sensor categories). Each asset features a manually designed, functional physical control panel that mirrors the capabilities of its digital API.
  • Shared State: Whether a robot presses a physical "Power" button or sends an OnOff command via the API, the internal state of the simulated device updates identically, affecting both the physical indicators (like LEDs) and the API return values.
  • Ambient Evolution: Device states actually affect the environment. A running oven increases room temperature; a light increases illuminance.
Figure 2. Architecture of PAIR. Agents interact with the physical simulation through physical actions and with a Matter-based API server through structured API calls. Both channels are coupled through a shared device state.

Datasets

To ensure the benchmark reflected real-world needs, the authors conducted a survey of 108 participants, collecting 1,443 unique user requests (e.g., "Put the towel in the washer and start the normal cycle").

Figure 4. Survey statistics. Survey overview and request characteristics (left), distribution of required skills under physical-only control (middle), and number of skills per request (right). The middle chart counts each skill type at most once per request; the right chart counts each repeated physical action, such as pressing buttons for two settings, and each perception skill once.

From this data, they constructed:

  • A 40-task benchmark: Every task requires at least one physical step (like opening a door) that cannot be done via API, but also includes steps that could be handled digitally.
  • 340 Human Demonstrations: Collected via teleoperation using the XLeRobot mobile manipulator to provide a baseline for training Vision-Language-Action (VLA) policies.

Key Insights and Ideas

  • The Physical Bottleneck: The most striking "aha moment" is that APIs don't help if you can't get to the device. Even with API access, robots failed because they couldn't navigate to the right room or grasp the handle of a door. Physical execution remains the primary hurdle for household autonomy.
  • API vs. Vision: High-level planners (like Qwen-VL) tended to choose API calls based on visual clarity rather than efficiency. If the robot was too far away to "read" the physical panel, the planner switched to the API. Interestingly, they were largely indifferent to API latency, even when delays were simulated.
  • The "Hybrid" Reality: The survey revealed that 60.4% of user requests cannot be solved by APIs alone. This confirms that the future of domestic robotics isn't just "software" or "hardware," but a tight integration of both.

Key Results

The researchers evaluated two state-of-the-art VLA policies ($\pi_{0.5}$ and GR00T N1.7) alongside various high-level planners.

  • Zero Success: None of the fine-tuned VLA policies completed a single evaluation task (0% Task Success Rate).
  • API Impact: While API access tripled the success rate of device operation subtasks (rising from ~11% to ~36%), it did nothing to solve the failures in navigation and grasping, which accounted for over 90% of failed rollouts.
  • Planning Limits: Even when the robot's physical body was replaced with a "perfect oracle" (scripted movements), the best high-level planner (GPT-6 Luna) only successfully completed 57.3% of its plans, highlighting significant gaps in current LLM-based reasoning for long-horizon household tasks.

Limitations and Future Work

The authors acknowledge that their demonstration dataset (340 episodes) is relatively small for training robust VLA policies, which likely contributed to the low success rates. Furthermore, the survey was constrained to the specific device categories available in PAIR.

Future work suggests expanding the diversity of tasks and improving the "common sense" reasoning of planners to better balance the trade-offs between physical interaction and digital control—such as understanding that checking a status via API is always faster than walking across a house to look at a screen.

Summary updated

Abstract

Household robots often need to operate appliances, and smart appliances offer both control panels and device APIs. Existing embodied AI simulation environments, however, model appliances solely as physical objects. We introduce PAIR, a smart-home simulation environment grounded in the Matter protocol~\footnote{https://csa-iot.org/all-solutions/matter/}, the industry standard for smart-home devices. PAIR provides 84 smart-device assets whose functional control panels and APIs act on a shared device state, with ambient conditions that evolve in response to both. From a survey of 1,443 requests, we construct a benchmark of 40 tasks that all require object manipulation yet remain solvable without API access, with 340 human demonstrations for policy training. Whether run alone, with a high-level planner, or with an API-enabled planner, two fine-tuned VLA policies complete none of the 120 evaluation episodes. Robots rarely get far enough to use the API: even with oracle plans and API access, over 90\% of failed rollouts stall at navigating, grasping, or operating a door. With a perfect oracle in place of the robot's body, the best planner still gets through only 57.3\% of its plan. When choosing between pressing a button and calling an API, several planners go by whether they can read the panel, not by API latency.\footnote{We will release our code and data upon publication.}
BibTeX
@misc{pyo2026pair,
  author = {Junseong Pyo and Hyeongjin Kim and Jongwook Han and Seungwon Kook and Sungjib Lim and Yohan Jo},
  title = {{PAIR: Bringing Household Robots into the Smart Home}},
  year = {2026},
  eprint = {2611.23070},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23070},
}
Figure