Large Language Models (LLMs) often struggle with "multi-hop" questions—queries that require connecting disparate pieces of information to reach a conclusion. For example, answering "Which city hosted the Olympics in the birth year of the author of The Great Gatsby?" requires identifying the author (F. Scott Fitzgerald), his birth year (1896), and then the Olympic host city for that year (Athens).
When a model attempts to solve this in a single pass, it often falls victim to "semantic mismatching" or the "lost in the middle" phenomenon, where the retriever fails to find the relevant documents because the query is too complex or ambiguous. To solve this, a growing body of research focuses on Query Decomposition: the process of breaking a complex, multi-faceted query into a sequence of simpler, atomic sub-questions.
The Divide and Conquer Mechanism
The core intuition behind query decomposition is that while a single complex query is a poor search term, its constituent parts are highly effective. By treating a long-form query as a structured plan rather than a single string, models can navigate external knowledge bases more precisely.
Researchers like Dheeru Dua and Sameer Singh introduced Successive Prompting, a framework that iteratively breaks down questions. Instead of guessing the final answer, the model asks, "What is the first thing I need to know?" It solves that sub-question, uses the answer to formulate the next sub-question, and repeats the process until the decomposition module determines no further steps are needed.
This "divide-and-conquer" strategy is not just for text. In the realm of Knowledge Graphs (KGs), Chao Wang and colleagues developed ROG, which decomposes logical queries into elementary 1-hop projections. By solving a multi-step problem as a series of manageable tasks, ROG outperforms traditional embedding-based methods by significant margins.
Sequential vs. Parallel Decomposition
Research generally splits decomposition into two architectural patterns:
- Sequential/Iterative Reasoning: The answer to sub-question $A$ is required to form sub-question $B$. This is essential for dependent hops. Weizhi Fei and Xueyan Niu demonstrate this in Retrieval Meets Reasoning, where they treat long texts as malleable knowledge bases, iteratively "editing" facts into the reasoning process as sub-questions are resolved.
- Parallel/Facet Decomposition: Some queries are complex because they are multifaceted rather than sequential. Reinald Kim Amplayo and Shashi Narayan use Query Refinement Prompts to identify different "facets" or interpretations of an ambiguous question simultaneously. By generating these sub-components first, the model creates a content plan that guides the synthesis of a comprehensive answer.
Improving Retrieval Precision
Decomposition acts as a bridge between the high-level intent of a user and the low-level indexing of a retriever. Youmin Ko and Sung Jong Seo show in CoopRAG that "unrolling" questions into masked reasoning chains can increase retrieval recall by over 35%. The structured sub-questions provide a richer signal for the retriever than a short, ambiguous natural language query.
Similarly, L.X. Ye and Linhao Ye introduced Q-DREAM, which optimizes the semantic space for retrieval by resolving dependencies between sub-questions. This allows the system to be 6x faster than iterative baselines by knowing which parts of the query can be fetched in parallel.
Reducing Hallucination through Verification
A major benefit of decomposition is interpretability. When a model fails on a complex question, it is often unclear where the logic broke. By decomposing the query, researchers can insert verification steps at each "hop."
Yang Zhao and Chengxiao Dai utilize this in AGENTICT2S, a multi-agent framework that reasons across disconnected knowledge graphs. By decomposing queries into subgoals and employing a two-stage verification process, they prevent "semantic drift"—the tendency of a model to wander off-topic as it moves through a reasoning chain.
This modularity also allows for specialized tool use. As Alon Jacovi and Mor Geva note in their evaluation of tool-assisted generation, recursive decomposition allows a model to route specific sub-questions to the most appropriate tool—sending a math sub-problem to a calculator and a factual sub-problem to a search engine.
Why It Works: The Reasoning Gap
The fundamental reason decomposition works is that it reduces the reasoning gap. LLMs are excellent at "single-hop" tasks (e.g., "When was Fitzgerald born?") but struggle with the compositionality of those tasks.
By explicitly mapping a complex query to a sequence of atomic operations—whether through logic-guided decomposition (ORACLE) or syntactic splitting (GRIEVER)—researchers ensure that the model is always operating within its "comfort zone" of high-confidence, single-step reasoning. This transforms the problem from a "black-box" generation task into a transparent, verifiable execution plan.