Abstract
Forgetting is inevitable in human memory. Recently, multimodal embedding models have been proposed to vectorize multimodal reality into a unified embedding space. Once generated, these embeddings allow mobile users to quickly retrieve relevant information, effectively augmenting their memory. However, as the model's capacity increases, its resource consumption also rises. The resulting slow throughput and significant computational resource requirements hinder its deployment on mobile devices. In this paper, we present Reminisce, an efficient on-device multimodal embedding system that enables high-throughput embedding and precise retrieval on resource-constrained mobile devices. The core design draws inspiration from the memory functions of the human brain, utilizing coarse-grained embeddings to identify likely candidates, which are then refined through query-driven fine-grained retrieval. A series of algorithm-hardware orchestrated optimizations automatically navigates this process and strengthen the embedding quality. Experiments show that Reminisce provides high-quality embedding representation with high throughput while operating silently in the background with negligible memory usage and reduced energy consumption.
Introduction: The Mobile Resource Wall
Modern mobile devices serve as digital chronicles of our lives, yet searching through this history effectively remains a challenge. Multimodal Embedding Models (MEMs) offer a solution by transforming diverse data—photos, videos, and notes—into a unified mathematical space. However, running these models on mobile hardware presents a significant "resource wall." Standard MEMs are often too slow to keep up with real-time data streams and consume excessive power. As illustrated in Figure 4, standard MEM implementations (FP32 and INT4) demand significantly higher power than typical mobile activities like video playback or gaming, which can reduce a device's usable time to just a few hours.
Reminisce: System Architecture and Methodology
Reminisce overcomes these limitations through a coarse-to-fine processing philosophy, mimicking the human brain's hierarchical recognition process. The system architecture, shown in Figure 2, bifurcates the workload into continuous background embedding ("Offline remembering") and infrequent foreground retrieval ("Online recall").
The system employs three primary hardware-software co-designs:
- Data-aware Pre-exit Predictor: A lightweight MLP analyzes intermediate activations to determine the earliest layer where an embedding is sufficiently accurate, allowing the system to skip redundant computations.
- Progressive LoRA Healing: To prevent accuracy degradation from early exits, Low-Rank Adaptation (LoRA) is used to fine-tune the model, ensuring shallow embeddings retain necessary semantic information.
- Speculative Fine-grained Retrieval: During search, coarse embeddings filter the database to a small candidate set. The full model is then applied only to these candidates, ensuring high precision with minimal overhead.
Evaluation and Key Results
The system was evaluated across multiple modalities using benchmarks like COCO, FLICKR, and CLOTHO, along with a custom Twitter Meme Dataset (803 images and descriptions) to simulate real-world usage.
Key Findings: * Throughput and Efficiency: Reminisce achieves a 12.4x average throughput improvement and up to 29x energy savings compared to standard layer-wise execution. * Memory Footprint: By optimizing execution, the system maintains a memory footprint of less than 200MB RAM, a 7x reduction over naive implementations. * Real-world Performance: In user studies, Reminisce successfully embedded 100% of daily data, whereas standard systems failed to keep pace and required multiple daily recharges.
Discussion and Future Work
While Reminisce significantly improves embedding efficiency, query latency remains around 1.2 to 1.5 seconds due to the fine-grained retrieval step. Future work will explore long-term memory management, such as intelligent "forgetting" or compression of older embeddings, to manage storage costs estimated at ~10GB per year. Additionally, while the system is highly optimized, peak performance remains dependent on modern mobile NPUs and GPUs found in flagship devices.
Summary updated