Abstract
We open-source MiMo-VL-Miloco-7B and its quantized variant MiMo-VL-Miloco-7B-GGUF, a pair of home-centric vision-language models that achieve strong performance on both home-scenario understanding and general multimodal reasoning. Built on the MiMo-VL-7B backbone, MiMo-VL-Miloco-7B is specialized for smart-home environments, attaining leading F1 scores on gesture recognition and common home-scenario understanding, while also delivering consistent gains across video benchmarks such as Video-MME, Video-MMMU, and Charades-STA, as well as language understanding benchmarks including MMMU-Pro and MMLU-Pro. In our experiments, MiMo-VL-Miloco-7B outperforms strong closed-source and open-source baselines on home-scenario understanding and several multimodal reasoning benchmarks. To balance specialization and generality, we design a two-stage training pipeline that combines supervised fine-tuning with reinforcement learning based on Group Relative Policy Optimization, leveraging efficient multi-domain data. We further incorporate chain-of-thought supervision and token-budget-aware reasoning, enabling the model to learn knowledge in a data-efficient manner while also performing reasoning efficiently. Our analysis shows that targeted home-scenario training not only enhances activity and gesture understanding, but also improves text-only reasoning with only modest trade-offs on document-centric tasks. Model checkpoints, quantized GGUF weights, and our home-scenario evaluation toolkit are publicly available to support research and deployment in real-world smart-home applications.
Introduction: The Challenge of Context-Aware Homes
Current smart-home systems often rely on rigid "if-this-then-that" rules or cloud-based AI that raises privacy and latency concerns. To move toward truly "context-aware" homes, we need models that can run locally on edge devices and understand complex human behaviors—like knowing the difference between someone reading a book versus playing on a phone.
However, there is a classic trade-off in AI: specialization vs. generality. When a model is fine-tuned to be an expert in one niche (like home gestures), it often suffers from "catastrophic forgetting," losing its ability to perform general tasks like solving math problems or reading documents. Xiaomi’s researchers set out to build a model that masters the home environment without losing its "general intelligence."
Methodology: Specialized Training for General Intelligence
The researchers built MiMo-VL-Miloco-7B on top of the existing MiMo-VL-7B backbone, using a sophisticated two-stage training approach designed to mitigate forgetting while maximizing domain expertise.
1. Supervised Fine-Tuning (SFT) with Chain-of-Thought
Instead of just teaching the model to output labels (e.g., "Reading"), the team utilized Chain-of-Thought (CoT) supervision. This forces the model to "think" through visual cues before reaching a conclusion. To make this practical for edge devices, they also introduced token-budget-aware reasoning, training the model to provide concise, direct answers when resources are tight.
2. Reinforcement Learning (RL) via GRPO
To recover general reasoning skills lost during home-specific fine-tuning, the team applied Group Relative Policy Optimization (GRPO). Unlike standard RL methods that require a complex "critic" model, GRPO estimates gains by comparing a group of outputs for the same query. This stage focused on "healing" the model's performance in video understanding, GUI grounding, and STEM reasoning.
Experimental Setup and Datasets
The model was trained and evaluated on a mix of proprietary and public data:
- Home-Scenario Data: A strict internal collection pipeline involving volunteers performing daily activities (Watch TV, Reading, Play Esports, Workout) and hand gestures (Thumbs Up, V Sign, OK, Shaka Sign, Open Palm).
- General Multimodal Data: The MiMo-VL corpus, covering VQA, image grounding, and OCR.
- RL Benchmarks: For the reinforcement stage, they used datasets like VTG-IT and LongVid for video dynamics, and various STEM-related queries for mathematical reasoning.
Performance Analysis
MiMo-VL-Miloco-7B significantly outperformed much larger closed-source and open-source models across both specialized and general benchmarks.
Home Mastery
As shown in the evaluation results, the model achieved leading F1 scores across the board, notably hitting 98.3 on "Watch TV" and 99.2 on "Play Esports." In gesture recognition, it crushed baselines on the difficult "Shaka Sign," outperforming the next best model by over 18 points.
General Prowess and Edge Readiness
Despite its specialization, the model scored 55.7% on MMMU-Pro, beating larger models like Gemma-3-27B-IT. It also set new state-of-the-art scores for its size on video benchmarks like Video-MMMU (63.6%). One of the most interesting findings is that teaching a model how to reason about a scene (via CoT) doesn't just help with logic—it actually makes the model better at basic visual recognition of gestures and activities.
Conclusion and Future Work
While the model is a powerhouse in the home, the authors noted a slight regression in document-centric tasks (like OCR and chart reading) compared to the base model. This is a side effect of prioritizing home-scenario data. Moving forward, the team plans to explore multi-objective optimization to further close the gap in document understanding and integrate more sensory modalities, such as audio and millimeter-wave signals, to create a more holistic "sensory organ" for the smart home.
Summary updated