Scaling Reasoning Chains Beyond Two Hops

23 concepts22 papers3 proposals

Most current progress in multi-hop reasoning has been defined by a "two-hop" paradigm. Benchmarks like HotpotQA and QASC require models to find a "bridge" entity to connect two disparate facts. While Large Language Models (LLMs) have become adept at these two-step inferences, they face a steep performance cliff when asked to scale to three or more reasoning steps. This limitation isn't just a matter of data; it reflects fundamental challenges in how models retrieve information, maintain logical coherence, and utilize their internal layers.

The Two Hop Wall

The research community has historically focused on shallow reasoning depths. Yang et al. and Khot et al. established the foundation for multi-hop QA by requiring models to compose two sentences or articles to reach an answer. However, as Khot et al. later noted, the retrieval methods designed for these tasks are often specialized for exactly two steps and do not readily scale to $N$-hop scenarios.

When moving beyond two hops, two primary issues emerge:

  1. Error Accumulation: In iterative retrieval models, such as those explored by Lee et al., each additional hop increases the risk of retrieving irrelevant information, which then poisons the rest of the reasoning chain.
  2. Structural Inconsistency: As the number of hops increases, the difficulty of maintaining a coherent logical structure grows significantly. Jhamtani et al. argue that models struggle to identify valid reasoning chains as the inference task becomes more complex.

The Sequential Reasoning Pathway

To understand why models fail at deeper reasoning, researchers have begun looking inside the "black box." Biran et al. identified a "sequential reasoning pathway" within LLMs. For a two-hop query, the model typically resolves the first hop (the bridge entity) in its early layers and the second hop in its later layers.

A critical failure mode discovered by Biran et al. is that models simply "run out of layers" to complete the reasoning process for more complex queries. If a task requires four hops but the model's internal pathway only has room for two or three sequential computations, the logic breaks down. Similarly, Li et al. found a "compositionality gap" where models fail to properly generate or leverage intermediate implicit facts in their middle layers.

Strategies for Depth Scalability

Researchers are proposing several architectural and procedural shifts to break through the two-hop limit:

Internal Signal Extraction

Rather than forcing the model into a slow, multi-step loop of generating text queries, Linjiaen et al. propose L-RAG. This method extracts "next-hop" search signals directly from the hidden states of the LLM during a single forward pass. This allows the model to "look ahead" and retrieve necessary information for subsequent hops without waiting for the final output layer.

Hierarchical and Latent Planning

To maintain coherence over long chains, some researchers suggest treating intermediate steps as hidden variables. Qi et al. introduced PLAR, a framework that uses hierarchical planning to learn complex reasoning from standard datasets without needing explicit labels for every sub-question. By planning the reasoning path globally, the model is less likely to drift off-course during a long chain.

Recursive and Back-Attention

Standard attention mechanisms are often linear. Yu et al. propose Back Attention, which allows the model to perform recursive or back-and-forth passes. This structural fix helps the model revisit earlier parts of the reasoning chain to refine its understanding as new facts are uncovered in deeper hops.

Reasoning in the Wild

The challenge of scalability extends beyond simple text QA. In document-level relation extraction, Zeng et al. and Ng et al. highlight that current systems struggle to resolve dependencies that span multiple intermediate entities or documents. Sun et al. introduced CodRED to push models toward "cross-document" reasoning, where the facts required for a single inference might be scattered across an entire corpus, requiring a chain of $N$ hops to synthesize.

As Talmor et al. demonstrate with MultiModalQA, these chains often need to cross modalities—hopping from a table to an image to a text snippet. The ultimate goal of this research direction is to move away from fixed, two-step templates toward architectures that can dynamically extend their reasoning depth to match the complexity of the real world.

Key Foundational Work

HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (2018) — contributed 1 concept elements to this direction.
Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA (2019) — contributed 1 concept elements to this direction.
Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_reasoning_chain_scalability_03abcdb9,
  title = {reasoning chain scalability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reasoning-chain-scalability/rdr_03abcdb94a5c46509ed3b7689c3f572c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_reasoning_chain_scalability_03abcdb9_v1,
  title = {reasoning chain scalability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reasoning-chain-scalability/rdr_03abcdb94a5c46509ed3b7689c3f572c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake