LMMR: Learned Multigranular Memory Retrieval for Small Language Model Agents

Pengfeng Song, Yunkai Zhuang, Liye ShiarXiv 2026 · arXiv

TL;DR

LMMR (Learned Multigranular Memory Retrieval) allows small language model (SLM) agents to efficiently solve multi-hop questions by choosing not just which documents to retrieve, but also the right granularity (paragraphs vs. sentence fragments). By prioritizing information density and conditional relevance, LMMR improves answer accuracy by up to 4.5 F1 points while simultaneously reducing the token footprint by roughly 25%.

What Problem Does This Paper Address?

Small language models (SLMs) are increasingly used as autonomous agents, but they face a major bottleneck: a limited "context window" or token budget. In multi-hop reasoning tasks—where an answer must be pieced together from multiple sources—standard retrieval methods often fail.

The authors identify two main issues:

  1. Irrelevant "Filler": Retrieving full paragraphs often brings in "background noise" that consumes precious token space without adding value.
  2. Redundancy: Standard retrievers might pick two highly-ranked documents that repeat the same fact, leaving no room for the second piece of the puzzle needed to bridge the gap to the answer.

Methods

LMMR introduces a "multigranular" approach to memory. Instead of just indexing paragraphs, it links paragraphs to their constituent sentence fragments.

The core of the system is a Conditional Reranker. Unlike a standard retriever that scores a document based solely on the query, LMMR’s reranker asks: "Given the information I already have (the 'anchor' passage), what does this new candidate add?"

Key technical highlights include:

  • Source-Consistent Learning: The model is trained using pairs of coarse (paragraph) and fine (sentence) views. If a single sentence provides the same missing support as the full paragraph, the model is taught to prefer the shorter "fine" view.
  • Budgeted Selection: An algorithm (Algorithm 1 in the paper) that ranks sources and then expands them into fine-grained views to find the best combination that fits within a strict token budget (e.g., 400 tokens).
  • Overlap Penalty: The system explicitly penalizes lexical overlap with the anchor text to ensure the retrieved evidence is complementary rather than repetitive.
Figure 2: LMMR illustrated through concrete memory views. (a) A source paragraph and its first-sentence view retain the same father relation. Under the same question and anchor, both supply new support; relevant non-supporting and redundant examples supervise the shared conditional reranker. (b) The reranker first ranks sources, then compares expanded sentence views with the paragraphs still available. Selection merges the anchor and one additional view within the token budget. Blue marks corresponding support, ellipses abbreviate displayed text, and dashed arrows show parameter sharing.

Datasets

The authors evaluated LMMR on three major multi-hop question-answering benchmarks:

  • HotpotQA: Focuses on "bridge" and "comparison" questions requiring two pieces of evidence.
  • 2WikiMultihopQA: Involves complex reasoning chains and entity linking.
  • MuSiQue: A challenging dataset designed to minimize "shortcuts" in multi-hop reasoning.

Testing was performed across eight different SLMs, ranging from 0.8B to 4B parameters (including Qwen, Gemma, and Phi-4).

Key Insights and Ideas

  • Density is King: For SLMs, the quality of tokens matters more than the quantity. By stripping away the fluff, LMMR provides a "higher-signal" input that makes it easier for the model to find the correct answer.
  • The "Joe Buck" Example: In a classic failure case (Figure 1), standard retrieval might provide two full paragraphs about Joe Buck and Bob Buck. The SLM, overwhelmed by text, might just output the father's name. LMMR selects one paragraph and one specific sentence, fitting both into half the space and allowing the model to correctly identify the "St. Louis Cardinals" as the team.
Figure 1: The same question and source records yield different small-model answers. (a) Query expansion supplies two paragraphs (235 tokens); the 2B model returns the father’s name. (b) LMMR uses the Bob Buck paragraph followed by a Joe Buck sentence (122 tokens), and the same model returns the correct team. Blue text marks supporting content; grey lines abbreviate background. Bars show evidence length on the same 400-token scale. Answers and lengths are from the recorded HotpotQA case.
  • Conditional Logic: Retrieval shouldn't be a static list. By making the second retrieval step dependent on the first, the system acts more like a human researcher who looks for "what's missing" rather than just "what's relevant."

Key Results

  • Accuracy Gains: On HotpotQA, a frozen 2B parameter model achieved 61.47 F1, a gain of 3.06 points over standard query expansion. On MuSiQue, the gain was even higher at 4.55 points.
  • Efficiency: LMMR used 23.1% to 26.9% fewer tokens than standard methods. It even outperformed dedicated compression tools like LLMLingua-2 when compared at similar token lengths.
  • Universal Benefit: The improvements held true across all eight small models tested, suggesting that LMMR is a robust "plug-and-play" upgrade for SLM-based agents.

Limitations and Future Work

  • Selection Depth: The current implementation focuses on selecting an anchor plus one additional node ($K_e = 2$). While effective for 2-hop questions, more complex 4-hop or 5-hop questions might require extending this logic to longer chains.
  • Fixed Budgets: The study primarily focused on a 400-token budget. While they performed a sweep of different budgets, the optimal balance between granularity and context might shift as SLM context windows continue to grow.
  • Inference Overhead: Conditional reranking adds a computational step compared to simple top-K retrieval, though the authors note that the reduction in prompt encoding time on edge devices (like the Jetson Orin) can offset this.

Summary updated

Abstract

Small language model agents have limited space to read the memory needed for multi-hop questions. Relevant facts are often scattered across records whose surrounding text consumes much of that space. We propose LMMR, a learned multigranular memory retrieval method that selects both the next source and how much of it to read. LMMR links paragraphs and sentence fragments to their original sources and pairs views that supply the same missing support during training. Relevant distractors and repeated information provide contrasting examples. At inference, the reranker scores what each candidate adds to an anchor passage, and a selector chooses the evidence within a token budget. On 22,398 official validation questions from HotpotQA, 2WikiMultihopQA, and MuSiQue, a frozen 2B answer model reaches 61.47, 43.26, and 30.58 F1 under a 400-token budget. The gains over query expansion are 3.06, 2.14, and 4.55 points, while memory input decreases by 23.1-26.9%. LMMR also uses 23.5-27.7% fewer input tokens than a coarse retriever trained on the same questions. Against that retriever followed by LLMLingua-2 at comparable input lengths, F1 gains reach 3.99 points and are significant on HotpotQA and 2WikiMultihopQA after Holm correction. All eight small answer models evaluated on HotpotQA improve over query expansion and MMR in both F1 and exact match.
BibTeX
@misc{song2026lmmr,
  author = {Pengfeng Song and Yunkai Zhuang and Liye Shi},
  title = {{LMMR: Learned Multigranular Memory Retrieval for Small Language Model Agents}},
  year = {2026},
  eprint = {2611.22425},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22425},
}
Figure