TL;DR
While large language models (LLMs) have vastly improved smart home assistants, they fail to handle "elliptical commands"—the natural conversational shortcuts humans use as shared context builds up over time. To bridge this gap, the authors introduce PEC-Home, a novel simulated dataset that benchmarks how well assistants interpret progressively ambiguous commands across multi-user and dynamic environments. Their findings reveal a sobering reality: even state-of-the-art models like GPT-4o experience severe performance drops when commands lose explicit detail.
What Problem Does This Paper Address?
When interacting with smart home assistants, current LLM frameworks operate under a flawed assumption: that user commands will remain explicit, static, and isolated. In reality, human communication relies heavily on progressive ellipsis. As you interact with someone (or something) over time, you naturally omit redundant details. For example, you might start with, "Hey, could you turn on the bedroom air conditioner and set the temperature to 26 degrees?" but days later, you'll simply say, "Can you handle the air conditioner?"
Evaluating this is tricky because it introduces two core challenges:
- Multi-User Preferences (Referential Ambiguity): Different family members have conflicting definitions of abstract concepts like a "comfortable temperature."
- Dynamic User Preferences (Intention Ambiguity): A single user's preferences change based on environmental factors (e.g., wanting "auto mode" when it's sunny vs. "sleep mode" when it's raining).
Current home assistants struggle to resolve these ambiguities because they lack the datasets and benchmarks needed to learn from long-term, context-heavy interactions.
Methods
The authors systematically evaluated 10 distinct open-source and closed-source LLMs (including the Qwen2.5 series, Gemma2 series, LLaMA-3.1-8B, Mistral-7B, and GPT-4o) using three primary prompting strategies:
- Zero-Shot Prompting: Evaluating the model's baseline ability without dialogue history.
- In-Context Learning (ICL): Providing illustrative examples of elliptical commands within the prompt.
- Retrieval-Augmented Generation (RAG): Equipping the models with a vector database to store and retrieve past dialogue history.
Additionally, they tested two mainstream tool-based autonomous agent frameworks (Sasha and SAGE) and explored the impact of domain-specific fine-tuning via LoRA on Qwen2.5-7B and Gemma2-9B.
Datasets
The paper introduces PEC-Home (Progressively Elliptical Commands Dataset for Smart Homes), which contains 1,780 dialogues involving 1,424 unique personas adapted from PersonaChat.
Unlike existing smart home datasets that rely on hard-coded rules or single explicit commands, PEC-Home structures each dialogue into a sequence of 4 chats that become progressively more elliptical:
- Level 1: All core components clear (Room, Device, Operation, Parameters).
- Level 2: Room omitted (spatial context assumed).
- Level 3: Operation and parameters omitted (descriptive modifiers dropped).
- Level 4: Only the device is specified.
Key Insights and Ideas
- Parameter Scale Isn't a Silver Bullet: While larger models perform better on explicit or mildly elliptical commands, this advantage vanishes when facing highly elliptical (Level 4) commands. Even GPT-4o underperforms smaller RAG-equipped models in extreme ellipsis scenarios.
- The "Noisy Memory" Penalty: In real-world deployments, databases get cluttered. The authors found that preloading even a small amount of irrelevant dialogue history into an assistant's memory drastically degrades its ability to resolve elliptical commands.
- Fine-Tuning Collapses on Shortcuts: Fine-tuning an LLM on smart home data significantly boosts its performance on explicit commands (Level 1), but completely collapses on highly elliptical commands (Levels 3 and 4). The models fail to generalize the underlying contextual logic.
- Divergent Failure Modes: The authors uncovered a fascinating split between Execution Accuracy (EA) and F1 scores. On explicit commands, EA is higher than F1 because models often "over-execute" (adding extra correct actions that lower F1 formatting compliance). On highly elliptical commands, EA drops below F1 because models take "shortcuts"—generating simple, partial matches that tick F1 boxes but fail to actually execute the user's true intent.
Key Results
The quantitative drop-off across ellipsis levels is stark. Performance was measured using Execution Accuracy (EA), which tracks whether the model successfully triggered the correct device state.
- Zero-Shot & ICL Limitations: Without history, all models fail completely at Levels 3 and 4 (accuracies often hovering near 0-2%).
- RAG Performance: RAG provides the best buffer, but still degrades. For instance, GPT-4o with RAG achieves 93.07% EA at Level 1 but drops to 55.71% EA at Level 4 in multi-user settings. DeepSeek-V3 shows a similar trend, dropping from 94.10% (Lv1) to 53.37% (Lv4).
- Tool-Based Agents Struggle: Mainstream agent architectures like SAGE and Sasha consistently underperformed compared to standard ICL/RAG setups on complete commands and failed to reliably manage memory retrieval when commands became vague.
Limitations and Future Work
The authors note two primary limitations:
- Context Length & Cost: Simulating a complex virtual home environment requires feeding detailed device and method descriptions into the prompt, leading to long contexts and high computational overhead.
- Lack of Real Production Data: Due to strict privacy constraints in smart homes, the dataset relies on LLM-generated dialogues. Although validated by a rigorous human-in-the-loop verification process (achieving 94.5% alignment with human linguistic patterns), real-world deployment data remains the gold standard.
Future work will likely focus on developing more robust memory-filtering mechanisms that can cleanly isolate relevant context from noisy household dialogue history.
Summary updated