TL;DR
Mobile AI Factory is an agentic system that automatically builds specialized, fine-tuned, and quantized small language models (SLMs) tailored for specific mobile devices. By combining LLM-driven data curation with real-world hardware profiling, it produces on-device models that often outperform massive frontier models while strictly adhering to a phone's unique latency, memory, and energy constraints.
What Problem Does This Paper Address?
Running language models directly on your phone is the "holy grail" for privacy and offline functionality. However, making this work is a nightmare for developers. A model that runs smoothly on a brand-new flagship might crash or crawl on a two-year-old budget phone.
Currently, creating a mobile-ready model requires "expert tuning": manually selecting a base model, fine-tuning it for a specific task, choosing a quantization level (like 4-bit vs. 8-bit), and testing it across various inference engines (like llama.cpp or MNN). Because hardware performance is highly unpredictable across different chips and drivers, this process is too slow and expensive to repeat for the thousands of smartphone models in the wild. The authors found that even high-reasoning LLMs like Claude Opus 5.5 cannot accurately predict which configuration will work on a specific phone just by looking at a spec sheet.
Methods
The Mobile AI Factory replaces the human expert with an agentic loop:
- Hardware Profiling: The system first measures the actual performance of various model configurations on the target phone. It benchmarks prefill/decode speeds, memory usage, and energy consumption. This "measured profile" acts as a filter, ensuring the search only considers models that can actually run within the developer's budget.
- Agentic Search: An "orchestrator" LLM (Claude 3.5 Sonnet) manages the training process. It decides on "interventions" for each iteration: tuning hyperparameters (like LoRA rank or learning rate), mining more real data, or using a "teacher" model (Qwen 3.5-35B) to generate synthetic training examples targeted at the model's current failures.
- Bucketed Search Strategy: To be efficient, the system groups 27 model variants into five size-based tiers. It starts with the smallest bucket and only "escalates" to larger models if accuracy stops improving.
- Deterministic Guardrails: While the LLM agent plans the search, fixed rules handle "rollbacks" (reverting to the best state if a change hurts accuracy) and decide when to move between tiers. This prevents the agent from wasting compute on dead ends.
Datasets
The system was evaluated on ten diverse tasks spanning extraction, classification, and generation:
- Extraction/Parsing: BC5CDR (biomedical entities), Calendar JSON extraction, TOPv2 (nested intent parsing), MultiCoNER (fine-grained entities), and BFCL (function calling).
- Classification/Generation: CLINC150 (151-way intent), SMS Spam detection, GoEmotions (28-way emotion), BEA-19 (grammar correction), and DialogSum (dialogue summarization).
Key Insights and Ideas
- The "Spec Sheet" Fallacy: You cannot reason your way to the best mobile configuration. As shown in Figure 2, a model can run 42× slower on one phone than another. More surprisingly, the "best" backend flips: GPU prefill is 5.4× faster than CPU on a OnePlus 15, but 7.4× slower on a OnePlus 8 Pro.
- Small Models Punch Up: With targeted fine-tuning, sub-billion parameter models can beat "frontier" models like Claude or DeepSeek on specific tasks. This is especially true for tasks with strict output formats (like JSON), where the specialized model learns the "rules" better than a general-purpose giant.
- Efficiency through Tiering: The bucketed search reached near-optimal accuracy on a biomedical extraction task while using 19× less GPU time than an exhaustive sweep of all model variants.
- Inference Engines Matter: The choice between engines like
llama.cppandMNNisn't just about speed; it affects accuracy. Different quantization recipes can cause an accuracy drop of up to 0.19 on the same weights.
Key Results
- Beating the Giants: On unseen test sets, Mobile AI Factory’s specialized models outperformed 5-shot frontier models on 5 out of 10 tasks (by up to 0.34 in F1 score) and matched them on 3 others.
- Strict Budget Adherence: In "hard" deployment scenarios (very tight memory or latency limits), the factory consistently found models that fit. Without real measurements, even the best reasoning models failed to pick a working configuration 60% of the time.
- Search Gains: Iterative search added a median of 8 points of accuracy on the smallest models compared to a simple first-pass fine-tune.
Limitations and Future Work
- Manual Harnessing: Developers still need to provide a task-specific verifier and scorer. The authors suggest that future agents could automate the creation of these evaluation harnesses.
- Static vs. Dynamic Profiling: Current profiling happens on an idle phone. In the real world, thermal throttling or other apps running in the background could change which configurations are feasible.
- Expanding the Pool: The search is currently limited to a predefined set of decoder models. Future versions could include encoder models (which are even smaller) or NPUs (Neural Processing Units) for faster inference.
- Profiling Speed: Profiling a phone is currently slow because the device must cool down between runs to ensure consistent measurements. Predicting performance from a subset of measurements is a promising research direction.
Summary updated