TL;DR
Conditional Retrieval Alignment (CoRA) is a gradient-free, hardware-friendly framework that turns frozen text and vision encoders into task-conditioned retrievers for on-device In-Context Learning (ICL). By replacing expensive backpropagation with closed-form linear algebra, CoRA achieves state-of-the-art exemplar selection while running efficiently within the strict memory and compute budgets of edge devices like a Raspberry Pi 5.
What Problem Does This Paper Address?
In-Context Learning (ICL) allows large language models to adapt to new tasks simply by providing a few demonstration examples (exemplars) in the prompt. However, the quality of these exemplars heavily dictates performance. On resource-constrained edge devices (like smartphones or single-board computers), selecting the right exemplars is a major challenge.
Standard retrievers (like BM25 or frozen embedding models) only look at input similarity, completely ignoring whether an example's output structure actually helps the model solve the target task. Conversely, existing task-aware retrievers require fine-tuning via backpropagation or making repeated, latency-heavy calls to a large model. These approaches are far too computationally expensive and memory-intensive for local, on-device deployment where data privacy and low latency are paramount.
Methods
The authors introduce CoRA, a fully gradient-free framework that constructs a task-conditioned retrieval space using paired candidate inputs and outputs via closed-form linear algebra.
The pipeline consists of three main stages:
- Representative Text Layer Selection: Instead of relying solely on the final layer of an encoder, CoRA uses Centered Kernel Alignment (CKA) to cluster and select a diverse subset of intermediate layers. This captures multi-level lexical, syntactic, and semantic features while minimizing residual redundancy.
- Task-Conditioned Retrieval Alignment: CoRA encodes paired candidate inputs and outputs from a local memory pool. It then maps the input representations to an output-derived conditioning space using closed-form ridge regression. This analytically isolates the input features most systematically associated with the target task's output regularities.
- Low-Rank Compression: Through spectral decomposition (SVD), CoRA extracts a compact, low-rank retrieval basis ($V_{1:r}$). At query time, candidate outputs are completely discarded; an unseen query input is simply projected onto this precomputed low-rank index for fast nearest-neighbor search.
To make it truly edge-compatible, the authors derive an exact two-pass streaming construction algorithm. By accumulating sufficient statistics in chunks, the peak working memory during index building becomes entirely independent of the candidate pool size ($n$). They also extend the framework to CoRA-M for multimodal tasks by incorporating visual features from a frozen vision encoder directly into the text-centric conditioning matrix.
Datasets
CoRA was evaluated across 14 diverse benchmarks:
- Textual Classification: SST-5 (sentiment), MRPC (paraphrase detection), QNLI (natural language inference), CMSQA, and HellaSwag (commonsense reasoning).
- Textual Generation: WebQs (open-domain QA), GeoQuery and NL2Bash (code generation), MTOP and SMCalFlow (semantic parsing).
- Multimodal Benchmarks: VQAv2, OKVQA, VizWiz (visual question answering), and MSCOCO (image captioning).
Downstream evaluation models included instruction-tuned Llama-3.2-1B, a 4-bit quantized MobileLLM-Pro, OpenFlamingo-3B, and Qwen3.5-2B/0.8B.
Key Insights and Ideas
- Outputs Anchor Task Specificity: Input similarity alone is a poor proxy for ICL utility. Aligning the retrieval space to the structural regularities of the outputs in your training/candidate pool fundamentally changes what counts as a "good" neighbor, leading to much better downstream model performance.
- Optimization Without Gradients: You don't need backpropagation to inject task-specific behavior into a retriever. Closed-form ridge regression provides an exact, analytical solution to align feature spaces instantly without altering the underlying encoder weights.
- Breaking the Memory Wall via Streaming: By restructuring the global ridge regression into chunk-wise matrix accumulations, the peak memory footprint during index construction scales with the feature dimension, not the dataset size. This allows edge devices to index massive candidate pools without running out of RAM.
Key Results
- State-of-the-Art Retrieval: CoRA consistently outperformed static retrieval baselines (BM25, SBERT, BERT) and prior optimization-based methods (MLSM) across all text and multimodal tasks.
- Massive Speedups on GPU: When compared to training-based retrievers like EPR or CEIL on a server, CoRA reduced the total task-adaptation-to-retrieval time by factors of 18.1x to 24.1x, while using a fraction of the memory.
- Successful Raspberry Pi 5 Deployment: In an end-to-end local deployment on a Raspberry Pi 5 (8GB RAM) using Qwen3.5-0.8B, CoRA achieved the highest accuracy across all evaluated tasks. Crucially, all training-based baselines (EPR, CEIL, TTF) suffered from Out-Of-Memory (OOM) errors and failed to run, while CoRA's chunked streaming successfully kept peak memory under 2.4 GB.
Limitations and Future Work
- Static vs. Dynamic Pools: The current streaming index construction assumes a fixed candidate pool during the two-pass compilation. Extending CoRA to handle continuously evolving, real-time streaming candidate pools without full re-indexing is an open engineering challenge.
- Hardware Acceleration: This study focused on a pure software implementation. However, because CoRA's pipeline relies entirely on standard dense linear algebra primitives (matrix multiplication, SVD, linear solvers), it is highly compatible with emerging In-Memory Computing (IMC) architectures, which could yield massive future gains in energy efficiency on edge hardware.
Summary updated