TL;DR
This paper introduces Recall, a novel system for on-device multimodal embedding that significantly boosts throughput and reduces energy consumption, making it practical for everyday mobile devices. By employing early exiting, progressive healing, and speculative retrieval, Recall achieves near state-of-the-art accuracy while operating efficiently with minimal resource demands.
What Problem Does This Paper Address?
Modern mobile devices generate vast amounts of multimodal data (text, images, audio, etc.). Multimodal Embedding Models (MEMs) can unify this data into a single embedding space for efficient retrieval, enabling powerful applications like personalized memory recall. However, current MEMs are computationally intensive, requiring significant processing power and energy, which limits their deployment on resource-constrained edge devices like smartphones. This leads to slow embedding speeds, high battery drain, and the inability to process all incoming data in real-time. The paper aims to bridge this gap by enabling efficient, on-device multimodal embedding without compromising user privacy or device performance.
Methods
Recall introduces three core techniques to achieve efficient on-device multimodal embedding:
- Data-aware Pre-exit Predictor: This novel component predicts the optimal "exit point" for an embedding computation early in the process. Instead of running the full deep neural network, the model can exit at an earlier layer if the embedding is deemed sufficiently accurate. This allows for batching and pipeline execution, significantly improving throughput.
- Progressive LoRA Healing (P-LoRA): To maintain accuracy despite early exits, Recall uses a parameter-efficient fine-tuning method called LoRA. P-LoRA tunes LoRA layers progressively, allowing for shared weights and efficient healing of exited branches. This approach optimizes the coarse-grained embeddings generated by early exits.
- Speculative Fine-grained Retrieval: This technique refines the coarse-grained embeddings by using a "live encoder" during the query phase. It involves speculative filtering, global verification, and fine-grained correction to ensure accurate retrieval even with early exits. This approach offloads the computationally intensive fine-grained embedding to the query time, where it's most needed.
Datasets
The paper uses the following datasets for evaluation:
- COCO: Text-Image retrieval.
- FLICKR: Image-Text retrieval.
- CLOTHO: Text-Audio retrieval.
- HARSMART: IMU retrieval.
- TWITTER: A real-world dataset of Twitter memes and descriptions, used for case studies to demonstrate performance on up-to-date topics.
Key Insights and Ideas
- Early Exiting is Key for Mobile MEMs: The paper highlights that traditional MEMs are not designed for the constraints of mobile devices. Early exiting, a technique more common in CNNs, is adapted and shown to be crucial for reducing computational load and enabling on-device deployment.
- Data Variability Matters: The "data-aware" aspect of the pre-exit predictor is critical. Different data samples have varying information content, and predicting exits based on this variability, rather than a fixed point, is essential for balancing accuracy and efficiency.
- Privacy is a Major Driver for On-Device MEMs: Offloading sensitive user data to the cloud for embedding poses privacy risks. On-device processing directly addresses this concern, making Recall a privacy-preserving solution.
- Trade-offs are Manageable: While early exiting can theoretically reduce accuracy, Recall's combination of techniques (healing and speculative retrieval) effectively mitigates this, achieving high accuracy with significantly reduced resource usage.
Key Results
- Throughput Improvement: Recall achieves an average of 14.9x improvement in throughput compared to standard MEMs.
- Energy Efficiency: Recall reduces energy consumption by an average of 13.1x compared to layerwise-executed baselines.
- Accuracy Preservation: Despite significant optimizations, Recall incurs less than 5% relative accuracy loss compared to full-sized MEMs.
- Query Latency: Query delays are kept under 5 seconds, making the system responsive for users.
- Resource Savings: On mobile devices, Recall significantly reduces memory footprint (up to 7.7x) and power consumption.
Limitations and Future Work
- Hardware Compatibility: While Recall is designed for mobile devices, the paper notes that current mobile inference engines may not fully support all modalities (e.g., IMU) or GPU optimizations.
- Dynamic Embedding Mismatch: The paper mentions that the output layer of the predictor is left untuned to mitigate dynamic embedding mismatch issues, suggesting this is an area for further exploration.
- Future Work: The authors plan to make their codebases and collected trace data public after acceptance, which will facilitate further research and development in this area. The paper also implicitly suggests that further exploration into optimizing the P-LoRA healing step and the speculative retrieval process could yield even greater improvements.
Summary updated