PEC-Home: Interpretation of Progressively Elliptical Commands in Smart Homes

Yingyu Shan, Zeming Liu, Silin Li, Boao Qian, Jiashu Yao, Yuhang Guo, Haifeng Wang

TL;DR

While large language models (LLMs) have vastly improved smart home assistants, they fail to handle "elliptical commands"—the natural conversational shortcuts humans use as shared context builds up over time. To bridge this gap, the authors introduce PEC-Home, a novel simulated dataset that benchmarks how well assistants interpret progressively ambiguous commands across multi-user and dynamic environments. Their findings reveal a sobering reality: even state-of-the-art models like GPT-4o experience severe performance drops when commands lose explicit detail.

What Problem Does This Paper Address?

When interacting with smart home assistants, current LLM frameworks operate under a flawed assumption: that user commands will remain explicit, static, and isolated. In reality, human communication relies heavily on progressive ellipsis. As you interact with someone (or something) over time, you naturally omit redundant details. For example, you might start with, "Hey, could you turn on the bedroom air conditioner and set the temperature to 26 degrees?" but days later, you'll simply say, "Can you handle the air conditioner?"

Evaluating this is tricky because it introduces two core challenges:

  1. Multi-User Preferences (Referential Ambiguity): Different family members have conflicting definitions of abstract concepts like a "comfortable temperature."
  2. Dynamic User Preferences (Intention Ambiguity): A single user's preferences change based on environmental factors (e.g., wanting "auto mode" when it's sunny vs. "sleep mode" when it's raining).
Figure 1: An example of PEC-Home. Multi-User Prefer- ences presents the referential ambiguity from conflicting "comfortable temperature" definitions between family members and Dynamic User Preferences indicates the intention ambiguity caused by environment changes.

Current home assistants struggle to resolve these ambiguities because they lack the datasets and benchmarks needed to learn from long-term, context-heavy interactions.

Methods

The authors systematically evaluated 10 distinct open-source and closed-source LLMs (including the Qwen2.5 series, Gemma2 series, LLaMA-3.1-8B, Mistral-7B, and GPT-4o) using three primary prompting strategies:

  • Zero-Shot Prompting: Evaluating the model's baseline ability without dialogue history.
  • In-Context Learning (ICL): Providing illustrative examples of elliptical commands within the prompt.
  • Retrieval-Augmented Generation (RAG): Equipping the models with a vector database to store and retrieve past dialogue history.

Additionally, they tested two mainstream tool-based autonomous agent frameworks (Sasha and SAGE) and explored the impact of domain-specific fine-tuning via LoRA on Qwen2.5-7B and Gemma2-9B.

Datasets

The paper introduces PEC-Home (Progressively Elliptical Commands Dataset for Smart Homes), which contains 1,780 dialogues involving 1,424 unique personas adapted from PersonaChat.

Unlike existing smart home datasets that rely on hard-coded rules or single explicit commands, PEC-Home structures each dialogue into a sequence of 4 chats that become progressively more elliptical:

  • Level 1: All core components clear (Room, Device, Operation, Parameters).
  • Level 2: Room omitted (spatial context assumed).
  • Level 3: Operation and parameters omitted (descriptive modifiers dropped).
  • Level 4: Only the device is specified.
Examples of the four ellipsis levels defined in PEC-Home

Key Insights and Ideas

  • Parameter Scale Isn't a Silver Bullet: While larger models perform better on explicit or mildly elliptical commands, this advantage vanishes when facing highly elliptical (Level 4) commands. Even GPT-4o underperforms smaller RAG-equipped models in extreme ellipsis scenarios.
  • The "Noisy Memory" Penalty: In real-world deployments, databases get cluttered. The authors found that preloading even a small amount of irrelevant dialogue history into an assistant's memory drastically degrades its ability to resolve elliptical commands.
  • Fine-Tuning Collapses on Shortcuts: Fine-tuning an LLM on smart home data significantly boosts its performance on explicit commands (Level 1), but completely collapses on highly elliptical commands (Levels 3 and 4). The models fail to generalize the underlying contextual logic.
  • Divergent Failure Modes: The authors uncovered a fascinating split between Execution Accuracy (EA) and F1 scores. On explicit commands, EA is higher than F1 because models often "over-execute" (adding extra correct actions that lower F1 formatting compliance). On highly elliptical commands, EA drops below F1 because models take "shortcuts"—generating simple, partial matches that tick F1 boxes but fail to actually execute the user's true intent.

Key Results

The quantitative drop-off across ellipsis levels is stark. Performance was measured using Execution Accuracy (EA), which tracks whether the model successfully triggered the correct device state.

  • Zero-Shot & ICL Limitations: Without history, all models fail completely at Levels 3 and 4 (accuracies often hovering near 0-2%).
  • RAG Performance: RAG provides the best buffer, but still degrades. For instance, GPT-4o with RAG achieves 93.07% EA at Level 1 but drops to 55.71% EA at Level 4 in multi-user settings. DeepSeek-V3 shows a similar trend, dropping from 94.10% (Lv1) to 53.37% (Lv4).
  • Tool-Based Agents Struggle: Mainstream agent architectures like SAGE and Sasha consistently underperformed compared to standard ICL/RAG setups on complete commands and failed to reliably manage memory retrieval when commands became vague.
Figure 3: Execution Accuracy of Sasha on Qwen2.5- 7B-Instruct across varying amounts of preloaded mem- ory in multi-user preferences and dynamic user prefer- ences scenarios. ‘Mem number’ indicates the amount of preloaded irrelevant memory.

Limitations and Future Work

The authors note two primary limitations:

  1. Context Length & Cost: Simulating a complex virtual home environment requires feeding detailed device and method descriptions into the prompt, leading to long contexts and high computational overhead.
  2. Lack of Real Production Data: Due to strict privacy constraints in smart homes, the dataset relies on LLM-generated dialogues. Although validated by a rigorous human-in-the-loop verification process (achieving 94.5% alignment with human linguistic patterns), real-world deployment data remains the gold standard.

Future work will likely focus on developing more robust memory-filtering mechanisms that can cleanly isolate relevant context from noisy household dialogue history.

Summary updated

Abstract

Recent advancements in Large Language Models (LLMs) have empowered home assistants with natural language interaction capabilities. However, current assistants overlook the progressive omission that occurs in human dialogue as shared context accumulates, leading to more elliptical expressions for efficient communication. Thus, current assistants still struggle to interpret such elliptical expressions accurately, which limits their effectiveness in real-world applications. In practical smart home scenarios, assistants face two major challenges caused by elliptical commands: (1) referential ambiguity caused by different environmental expectations among multiple users; and (2) intention ambiguity resulting from user preferences that evolve over time or change with the environment. To address these challenges, we introduce PEC-Home, the first simulated home dataset specifically designed for interpreting progressively elliptical commands in smart homes. Extensive experiments on various LLMs, including GPT-4o, show that existing home assistants struggle to execute user-intended operations based solely on elliptical commands. Even when equipped with tools for storing and retrieving user dialogue history, execution accuracy remains below that achieved with complete commands.}.
BibTeX
@inproceedings{shan2026pec,
  author = {Yingyu Shan and Zeming Liu and Silin Li and Boao Qian and Jiashu Yao and Yuhang Guo and Haifeng Wang},
  title = {{PEC-Home: Interpretation of Progressively Elliptical Commands in Smart Homes}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.999},
}

Loading related content...

Figure