Abstract
This paper introduces an unsupervised framework for learning to describe human activities within a smart home. By combining multimodal sensor data—capturing motion, location, and object interactions—with a Bayesian grammar model, the system achieves language grounding without the need for large-scale, manually labeled datasets. The approach addresses critical issues in personal robotics, including user privacy and environmental adaptability, by learning word meanings and grammatical structures directly from raw data and situational "teaching sentences."
Introduction
Modern video-to-text systems typically rely on Deep Learning models trained on vast, human-labeled datasets like ImageNet or MS COCO. While effective, this paradigm faces significant hurdles in the context of smart homes and personal robotics. First, privacy is a major concern; uploading sensitive daily life data for crowdsourced labeling is often unacceptable. Second, adaptability is limited, as pre-trained models struggle with unique home layouts or idiosyncratic user vocabulary. Finally, visual data alone often lacks the context necessary to distinguish subtle actions.
To overcome these challenges, the authors propose a system for unsupervised language grounding. This system learns the relationships between words and physical actions entirely from raw sensor inputs and teaching sentences provided in-situ, allowing for a personalized and privacy-preserving AI assistant.
Methods and System Architecture
The proposed system employs a hierarchical "bottom-up" architecture that transitions from raw sensor signals to abstract concepts and structured natural language.
Multimodal Sensing and Concept Learning
The system utilizes a "Smart House" environment equipped with diverse sensors to capture the nuances of human activity.
- mMLDA (Multilayered Multimodal Latent Dirichlet Allocation): This component acts as the primary categorization engine. It clusters multimodal inputs into three distinct concept types: Object (via RFID), Motion (via accelerometers), and Place (via laser range finders).
- Mutual Information (MI) for Word Grounding: The system calculates the MI between words in teaching sentences and inferred physical concepts. Words with high MI are linked to specific entities (e.g., "juice" to an object), while low-MI words are categorized as "functional words" (e.g., "the," "in") that provide grammatical structure.
Grammar and Sentence Generation
- Bayesian Hidden Markov Model (BHMM): While mMLDA identifies what is happening, the BHMM learns the grammar. It treats parts of speech as hidden states to model the typical sequence of concepts, such as an [Object] preceding a [Motion].
- Sentence Generation: To describe a new activity, the system samples concept sequences from the learned grammar and uses the Viterbi algorithm to find the most probable word sequence that aligns with both the observed sensor data and the linguistic model.
Experimental Setup
The study was conducted in a specialized Smart House environment with three subjects performing various daily tasks.
The dataset comprised 195 multimodal data points paired with 1,170 teaching sentences. Sensors included wearable cameras, laser range finders for positioning, RFID readers for object identification, and 3-axis gyro-accelerometers for motion capture.
Results
The proposed model was evaluated against a baseline lacking the Bayesian grammar component.
- Concept Selection: The system achieved 73.08% accuracy in identifying the concepts behind words, significantly outperforming the baseline (65.38%).
- Categorization Accuracy: Multimodal sensing proved highly effective, with 100% accuracy in object categorization and 95.96% in place categorization.
- Linguistic Quality: Automated BLEU scores and human evaluations confirmed superior performance. In human "Turing-style" tests, the system's sentences were rated as both grammatically and semantically correct approximately 45% of the time, nearly doubling the baseline's performance (23%).
Discussion and Conclusion
A key strength of the system is its ability to discover functional words—grammatical markers like "with" or "the"—by identifying their low correlation with specific physical sensors. However, limitations remain, particularly in motion sensing, which achieved only 52.53% accuracy due to the similarity between actions like "wiping" and "scrubbing."
Future work will explore non-parametric Bayesian models to allow the system to autonomously discover new categories of actions without predefined counts. Additionally, scaling the system to less controlled, "real-world" home environments remains a primary objective.
Summary updated