MUSE: A Heterogeneity-Aware Multimedia Search Engine for Mobile SoCs

Xinkui Zhao, Qingyu Ma, Yifan Zhang, Hengxuan Lou, Sai Liu, Chang Liu, Guanjie Cheng, Naibo Wang, Yueshen XuarXiv 2025 · arXiv

TL;DR

MUSE is a novel multimedia search engine custom-built for mobile Systems-on-Chip (SoCs) that solves the "performance paradox" of slow on-device vector search. By co-designing an NPU-centric asynchronous pipeline, a hardware-aligned index, and a workload-aware heterogeneous scheduler, MUSE delivers up to 7× faster index construction and 4.6× lower energy consumption while keeping smartphones safely below thermal throttling thresholds.

What Problem Does This Paper Address?

Next-generation mobile applications like personal AI agents and continuous lifelogging rely heavily on on-device Vision-Language Models (VLMs). To preserve user privacy and minimize latency, these applications encode multimedia streams (like photos, audio, and screen recordings) into high-dimensional feature vectors (often $\ge 1024$ dimensions) and store them locally for semantic similarity retrieval.

However, existing vector search systems are designed for server-class hardware and fail on mobile SoCs. When developers naively port state-of-the-art vector search algorithms like HNSW (Hierarchical Navigable Small World) to mobile Neural Processing Units (NPUs), they encounter a performance paradox: the NPU implementation actually runs up to 37× slower than an optimized CPU baseline.

Figure 3: Comparison of retrieval latency between mobile CPU and an unoptimized NPU across different indexing al- gorithms.

This degradation happens because graph-based algorithms rely on irregular memory access patterns and pointer-chasing, which clash with the regular, block-oriented execution models of NPUs. Furthermore, constant data type conversions (e.g., FP32 to FP16) and fragmented data movement create massive system overheads.

Figure 4: Latency breakdown of unoptimized multimedia vector search on a mobile NPU.

Methods

The authors introduce MUSE (MUltimedia Search Engine), an architecture-index co-design tailored specifically for the heterogeneous compute units of modern mobile SoCs (such as the Snapdragon 8 Elite).

Figure 2: The architectural blueprint of the Snapdragon 8 Elite Gen 5, highlighting heterogeneous compute units (CUs) and the shared memory hierarchy. Oryon-L denotes Qual- comm’s custom high-performance CPU core; the Hexagon NPU integrates the HVX vector unit and HMX matrix engine, backed by a tightly coupled memory (TCM).

MUSE achieves high-efficiency retrieval through three core techniques:

  1. NPU-Centric Asynchronous Pipeline: MUSE refactors vector retrieval into dense linear algebra operations. It utilizes a three-stage asynchronous pipeline that completely overlaps hardware DMA memory transfers, NPU matrix computations (via the Matrix Engine), and data layout adaptation (via the Vector Unit). It also leverages a zero-copy shared-memory design (dma-buf/ION) to eliminate OS-level memory duplication between the CPU and NPU.
  2. Hardware-Aligned Index Design: MUSE abandons graph-based indices in favor of an Inverted File Index (IVF). IVF decomposes retrieval into centroid routing and candidate scoring, which map cleanly onto dense General Matrix Multiply (GEMM) operations. Crucially, MUSE aligns the index parameters (like cluster counts) to match the NPU's native hardware tile geometries (multiples of 64), eliminating padding overhead.
  3. Workload-Aware Heterogeneous Scheduler: Instead of blindly offloading everything to the NPU, a dynamic scheduler routes tasks based on the hardware's ideal efficiency regimes. Latency-sensitive interactive queries go to the CPU; continuous background media ingestion is handled via CPU-GPU co-processing (leaving the NPU free for primary AI model inference); and heavy, periodic index rebuilds engage the CPU, GPU, and NPU simultaneously using a memory-efficient sliding-window model.
Figure 5: End-to-end heterogeneous scheduling and asynchronous NPU execution pipeline in MUSE. (a) Transfer orchestration using a sliding-window scheduler and zero-copy memory access. (b) Pack-transfer overlapping, hiding on-the-fly FP32→FP16 layout conversion behind DMA fetches. (c) Kernel-unpack overlapping, running GEMM computation concurrently with FP16→FP32 result reconstruction.

Datasets

MUSE was evaluated using two massive, high-dimensional embedding datasets ($d=1024$) generated via desktop GPUs to simulate real-world mobile AI workloads:

  • Text-to-Text Workload (Baseline): Derived from the HotpotQA dataset using the Qwen3-Embedding-4B encoder, scaling up to 889K vectors.
  • Image/Vision Workload (Multimedia): Derived from the MS-COCO dataset using the multimodal Qwen3-VL-Embedding-8B encoder, scaling up to 118.3K vectors.

The authors also evaluated scalability using synthetic feature spaces up to $d=4096$ to match next-generation Large Multimodal Models (LMMs).

Key Insights and Ideas

  • The Algorithmic Shift: On servers, graph-based indices (HNSW) are king due to their high recall and fast query times. On mobile SoCs, however, HNSW's massive memory footprint and irregular pointer-chasing lead to cache thrashing and memory exhaustion. IVF's contiguous vector storage is far more robust and accelerator-friendly.
  • Hiding the Layout Tax: NPUs prefer lower-precision formats like FP16, while mobile applications typically store embeddings in FP32. MUSE proves that you can hide the "layout tax" entirely by performing precision downcasting and matrix transpositions in-place on the NPU's vector processors while the matrix engine computes the previous tile's scores.
  • Co-designing Geometry: Hardware accelerators are rigid. A simple software adjustment—forcing the number of IVF clusters to be a multiple of the NPU's tile size (64)—prevents matrix fragmentation and yields dramatic, non-linear performance gains.

Key Results

Evaluated on commercial Snapdragon 8-series SoCs, MUSE consistently dominates traditional mobile vector search baselines across the Pareto frontier:

Figure 6: Query performance and index construction/rebuild latency for private on-device search: Recall@1 vs. QPS (left) and build time (right) across diverse multimedia database scales and hardware generations. MUSE consistently dominates the Pareto frontier.
  • Throughput & Speed: MUSE improves query throughput by up to 1.4× at matched recall, accelerates index construction by up to 7×, and delivers up to 6× higher vector insertion throughput under concurrent background streaming.
  • Physical Compliance: While CPU-bound baselines caused smartphones to spike near 50°C and trigger severe thermal throttling, MUSE's accelerator-native design capped peak device temperatures at a cool 38°C.
  • Energy Efficiency: By minimizing the active compute window, MUSE reduced total system energy consumption by up to 4.6× compared to HNSW and 2.3× compared to standard mobile IVF.

Limitations and Future Work

A primary limitation is hardware portability: MUSE's fine-grained DMA and instruction set architecture (ISA) controls rely on low-level, undocumented Qualcomm SDK interfaces. Other mobile chip vendors currently restrict NPU access to high-level, black-box model APIs, preventing similar optimizations on those platforms.

For future work, the authors plan to explore the integration of highly quantized vector storage formats (such as FP8 or INT4) directly into the pipeline, asynchronously unpacking them to FP16 within the NPU's tightly coupled memory to further compress the on-device memory footprint.

Summary updated

Abstract

On-device multimedia retrieval is vital for smartphones, enabling applications like cross-modal semantic search and multimodal personal AI agents. However, realizing efficient retrieval on mobile SoCs remains a critical systems challenge. Unlike text-based search, multimedia applications depend on high-dimensional cross-modal embeddings (often exceeding 1024 dimensions) and continuously expanding media streams (e.g., video lifelogs and ambient audio). Mobile devices must process these intensive workloads under strict latency, energy, and memory constraints. Existing vector retrieval systems, designed primarily for static server datasets, are poorly suited for the dynamic execution models of mobile SoCs, where interactive queries must coexist with continuous background media ingestion and index maintenance. To address these challenges, we propose MUSE, a MUltimedia Search Engine comprehensively co-designed for mobile SoCs. MUSE tackles these issues through an architecture-index co-design. It introduces a three-stage asynchronous execution pipeline that overlaps DMA transfers, NPU matrix computation, and hardware-vector data adaptation to overcome the memory wall of high-dimensional embeddings. Furthermore, it redesigns the Inverted File Index (IVF) and introduces workload-aware heterogeneous scheduling across the CPU, GPU, and NPU for hybrid interactive-and-ingestion workloads. Evaluated on Snapdragon 8-series SoCs using real-world multimodal datasets, MUSE improves query throughput by up to 1.4x at matched recall, achieves up to 7x faster index construction, and delivers up to 6x higher insertion throughput under concurrent streaming. Crucially, its accelerator-native design ensures strict physical compliance, capping peak device temperatures at 38°C and reducing total energy consumption by up to 4.6x compared to CPU-bound baselines.
BibTeX
@misc{zhao2025muse,
  author = {Xinkui Zhao and Qingyu Ma and Yifan Zhang and Hengxuan Lou and Sai Liu and Chang Liu and Guanjie Cheng and Naibo Wang and Yueshen Xu},
  title = {{MUSE: A Heterogeneity-Aware Multimedia Search Engine for Mobile SoCs}},
  year = {2025},
  eprint = {2511.19192},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.19192},
}
Figure