Ubiquitous Memory Augmentation via Mobile Multimodal Embedding System

Dongqi Cai, Shangguang Wang, Peng Chen, Zeling Zhang, Nicholas D. Lane, Mengwei Xu

TL;DR

This paper introduces Reminisce, an efficient on-device system for multimodal embedding that significantly boosts throughput and reduces energy consumption, making it practical for mobile devices. It achieves this by mimicking human memory's coarse-to-fine retrieval process, enabling users to recall vast amounts of personal data seamlessly and privately.

What Problem Does This Paper Address?

Mobile devices are constantly generating a wealth of multimodal data (text, images, audio, etc.) that could enrich our lives. However, efficiently retrieving this data is a major challenge. Current multimodal embedding models, while powerful, are too resource-intensive (computationally and energetically) to run effectively on resource-constrained mobile devices. This leads to slow performance, high battery drain, and limits their deployment for real-time, on-device applications, preventing users from leveraging their personal data as a true "memory palace."

Methods

Reminisce tackles this problem with a novel approach inspired by human memory:

  • Coarse-grained to Fine-grained Retrieval: Instead of processing data through the entire deep learning model for every query, Reminisce first generates "coarse-grained" embeddings. These are quicker to produce and act as a first pass to identify likely candidates. These candidates are then refined using a more computationally intensive "fine-grained" retrieval process, but only for the most promising matches.
  • Data-aware Pre-exit Predictor: A key innovation is a lightweight predictor that dynamically determines the optimal "exit point" for embedding a given piece of data. This means the model doesn't always need to run through all its layers, saving computation. This predictor is trained on intermediate embeddings and can adapt to different data types and modalities.
  • Progressive LoRA Healing (P-LoRA): To further optimize the early-exit strategy, Reminisce uses a parameter-efficient technique called LoRA (Low-Rank Adaptation). P-LoRA progressively heals the model's layers, allowing for efficient fine-tuning and reuse of embeddings across different exit points, which is crucial for maintaining accuracy.
  • Speculative Fine-grained Retrieval: To address potential accuracy drops from early exits, Reminisce employs a speculative approach. It filters candidates at multiple granularities and then refines them, ensuring that even with early exits, the final retrieval is precise.
  • Algorithm-Hardware Orchestration: The system integrates various optimizations, including pipeline execution and intermediate activation caching, to maximize throughput and minimize latency on mobile hardware.

Datasets

The paper evaluates Reminisce on several publicly available datasets across different modalities:

  • COCO: For text-image retrieval.
  • FLICKR: For image-text retrieval.
  • CLOTHO: For text-audio retrieval.
  • HARSMART: For IMU (Inertial Measurement Unit) data retrieval.

Additionally, a TWITTER meme dataset was used for a case study to demonstrate real-world applicability on unseen data. A user study also collected mobile application traces from eight users over a week to simulate daily usage scenarios.

Key Insights and Ideas

  • Human Memory as a Blueprint: The paper effectively demonstrates that mimicking the brain's hierarchical memory retrieval process can lead to significant efficiency gains in AI systems.
  • Early Exits are Not Enough: Simply exiting early from a deep model can hurt accuracy. Reminisce's success lies in combining early exits with intelligent healing and speculative refinement.
  • Data-Awareness is Crucial: The "data-aware pre-exit predictor" is a game-changer, allowing the system to adapt its computational effort based on the specific data being processed, rather than using a fixed strategy.
  • Privacy-Preserving AI: By enabling on-device processing, Reminisce addresses critical privacy concerns associated with offloading sensitive personal data to the cloud.

Key Results

  • Throughput Improvement: Reminisce achieves an average of 14.9x improvement in throughput compared to standard multimodal embedding models (MEMs).
  • Energy Efficiency: It reduces energy consumption by up to 18.2x compared to layerwise baselines and 13.1x on average.
  • Accuracy Preservation: Despite significant speedups, Reminisce maintains high retrieval accuracy, with only a less than 3% accuracy loss on the COCO dataset compared to full-sized MEMs.
  • Practical Deployment: The system demonstrates its practicality on various mobile devices, including smartphones and development boards, with minimal memory footprint.
  • Twitter Meme Retrieval: Reminisce achieved a 4x throughput improvement and used 5x less memory and 10x less energy than a naive MEM on a Twitter meme dataset.

Limitations and Future Work

  • Peak Memory Footprint: Reminisce has a slightly higher peak memory footprint compared to naive layer-wise baselines due to batching parallelism. However, the authors argue this is still within a practical range for modern devices.
  • Future Work: The paper suggests that mobile users can now efficiently index and recall multimedia content, fostering new use cases in personal assistants and health tracking. The authors also highlight the potential for Reminisce to be integrated into existing mobile applications without complex hardware modifications.

Summary updated

Abstract

<title>Abstract</title> Forgetting is inevitable in human memory. Recently, multimodal embedding models have been proposed to vectorize multimodal reality into a unified embedding space. The generated embeddings can be easily retrieved to help mobile users remember and recall information when needed. However, as the model’s capacity increases, its resource consumption also rises. The resulting slow throughput and significant computational resource requirements hinder its deployment on mobile devices. In this paper, we present Reminisce, the first efficient on-device multimodal embedding system that enables high-throughput and precise retrieval on resource-constrained mobile devices. The core design draws inspiration from the memory functions of the human brain, utilizing coarse-grained embeddings to identify likely candidates, which are then refined through query-driven fine-grained retrieval. A series of algorithm-hardware orchestrated optimizations automatically navigates this process and strengthens the embedding quality. Experiments show that Reminisce provides high-quality embedding representation with high throughput while operating silently in the background with negligible memory usage and reduced energy consumption.
BibTeX
@inproceedings{cai2025ubiquitous,
  author = {Dongqi Cai and Shangguang Wang and Peng Chen and Zeling Zhang and Nicholas D. Lane and Mengwei Xu},
  title = {{Ubiquitous Memory Augmentation via Mobile Multimodal Embedding System}},
  booktitle = {Nature Communications},
  year = {2025},
  doi = {10.1038/s41467-025-60802-5},
}
Figure