TL;DR
MUSE is a novel multimedia search engine custom-built for mobile Systems-on-Chip (SoCs) that solves the "performance paradox" of slow on-device vector search. By co-designing an NPU-centric asynchronous pipeline, a hardware-aligned index, and a workload-aware heterogeneous scheduler, MUSE delivers up to 7× faster index construction and 4.6× lower energy consumption while keeping smartphones safely below thermal throttling thresholds.
What Problem Does This Paper Address?
Next-generation mobile applications like personal AI agents and continuous lifelogging rely heavily on on-device Vision-Language Models (VLMs). To preserve user privacy and minimize latency, these applications encode multimedia streams (like photos, audio, and screen recordings) into high-dimensional feature vectors (often $\ge 1024$ dimensions) and store them locally for semantic similarity retrieval.
However, existing vector search systems are designed for server-class hardware and fail on mobile SoCs. When developers naively port state-of-the-art vector search algorithms like HNSW (Hierarchical Navigable Small World) to mobile Neural Processing Units (NPUs), they encounter a performance paradox: the NPU implementation actually runs up to 37× slower than an optimized CPU baseline.
This degradation happens because graph-based algorithms rely on irregular memory access patterns and pointer-chasing, which clash with the regular, block-oriented execution models of NPUs. Furthermore, constant data type conversions (e.g., FP32 to FP16) and fragmented data movement create massive system overheads.
Methods
The authors introduce MUSE (MUltimedia Search Engine), an architecture-index co-design tailored specifically for the heterogeneous compute units of modern mobile SoCs (such as the Snapdragon 8 Elite).
MUSE achieves high-efficiency retrieval through three core techniques:
- NPU-Centric Asynchronous Pipeline: MUSE refactors vector retrieval into dense linear algebra operations. It utilizes a three-stage asynchronous pipeline that completely overlaps hardware DMA memory transfers, NPU matrix computations (via the Matrix Engine), and data layout adaptation (via the Vector Unit). It also leverages a zero-copy shared-memory design (
dma-buf/ION) to eliminate OS-level memory duplication between the CPU and NPU. - Hardware-Aligned Index Design: MUSE abandons graph-based indices in favor of an Inverted File Index (IVF). IVF decomposes retrieval into centroid routing and candidate scoring, which map cleanly onto dense General Matrix Multiply (GEMM) operations. Crucially, MUSE aligns the index parameters (like cluster counts) to match the NPU's native hardware tile geometries (multiples of 64), eliminating padding overhead.
- Workload-Aware Heterogeneous Scheduler: Instead of blindly offloading everything to the NPU, a dynamic scheduler routes tasks based on the hardware's ideal efficiency regimes. Latency-sensitive interactive queries go to the CPU; continuous background media ingestion is handled via CPU-GPU co-processing (leaving the NPU free for primary AI model inference); and heavy, periodic index rebuilds engage the CPU, GPU, and NPU simultaneously using a memory-efficient sliding-window model.
Datasets
MUSE was evaluated using two massive, high-dimensional embedding datasets ($d=1024$) generated via desktop GPUs to simulate real-world mobile AI workloads:
- Text-to-Text Workload (Baseline): Derived from the HotpotQA dataset using the
Qwen3-Embedding-4Bencoder, scaling up to 889K vectors. - Image/Vision Workload (Multimedia): Derived from the MS-COCO dataset using the multimodal
Qwen3-VL-Embedding-8Bencoder, scaling up to 118.3K vectors.
The authors also evaluated scalability using synthetic feature spaces up to $d=4096$ to match next-generation Large Multimodal Models (LMMs).
Key Insights and Ideas
- The Algorithmic Shift: On servers, graph-based indices (HNSW) are king due to their high recall and fast query times. On mobile SoCs, however, HNSW's massive memory footprint and irregular pointer-chasing lead to cache thrashing and memory exhaustion. IVF's contiguous vector storage is far more robust and accelerator-friendly.
- Hiding the Layout Tax: NPUs prefer lower-precision formats like FP16, while mobile applications typically store embeddings in FP32. MUSE proves that you can hide the "layout tax" entirely by performing precision downcasting and matrix transpositions in-place on the NPU's vector processors while the matrix engine computes the previous tile's scores.
- Co-designing Geometry: Hardware accelerators are rigid. A simple software adjustment—forcing the number of IVF clusters to be a multiple of the NPU's tile size (64)—prevents matrix fragmentation and yields dramatic, non-linear performance gains.
Key Results
Evaluated on commercial Snapdragon 8-series SoCs, MUSE consistently dominates traditional mobile vector search baselines across the Pareto frontier:
- Throughput & Speed: MUSE improves query throughput by up to 1.4× at matched recall, accelerates index construction by up to 7×, and delivers up to 6× higher vector insertion throughput under concurrent background streaming.
- Physical Compliance: While CPU-bound baselines caused smartphones to spike near 50°C and trigger severe thermal throttling, MUSE's accelerator-native design capped peak device temperatures at a cool 38°C.
- Energy Efficiency: By minimizing the active compute window, MUSE reduced total system energy consumption by up to 4.6× compared to HNSW and 2.3× compared to standard mobile IVF.
Limitations and Future Work
A primary limitation is hardware portability: MUSE's fine-grained DMA and instruction set architecture (ISA) controls rely on low-level, undocumented Qualcomm SDK interfaces. Other mobile chip vendors currently restrict NPU access to high-level, black-box model APIs, preventing similar optimizations on those platforms.
For future work, the authors plan to explore the integration of highly quantized vector storage formats (such as FP8 or INT4) directly into the pipeline, asynchronously unpacking them to FP16 within the NPU's tightly coupled memory to further compress the on-device memory footprint.
Summary updated