Structural Generalization for Deeply Nested Sequences

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2138 papers221 conceptsfrozen 2026-10-04

While modern language models can generate fluent prose and solve complex coding tasks, they possess a hidden fragility: they struggle to process the recursive, tree-like structures that define human language and formal logic. This research direction investigates structural generalization—the ability of a model to apply learned rules to sequences that are more deeply nested or structurally complex than those encountered during training.

The Depth Barrier

In natural language, recursion allows us to embed phrases within phrases indefinitely (e.g., "The dog [that the cat [that the bird saw] chased] ran away"). While humans handle these hierarchies with relative ease, standard neural architectures face a "performance collapse" when nesting depth increases.

Yair Lakretz and colleagues demonstrated that causal Transformers, including models like GPT-2, perform below chance on recursive nested constructions once a phrase exceeds a certain length Causal Transformers Perform Below Chance on Recursive Nested Constructions, Unlike Humans. Similarly, Hitomi Yanaka has shown that models often master logical inferences at a specific depth but fail catastrophically when the same logic is applied to a structure just one level deeper Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language?.

This suggests that models are not learning abstract recursive rules. Instead, they appear to be learning "shallow" templates or statistical shortcuts that do not scale to the true productivity of language.

Lexical vs. Structural Generalization

A key insight in this field is the distinction between two types of generalization. Most current benchmarks measure lexical generalization—the ability to use a known word in a new position. Pretrained models like T5 and RoBERTa excel at this. However, Najoung Kim, Tal Linzen, and Paul Smolensky argue that this success is an illusion caused by "uncontrolled lexical exposure" Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Models.

When these models are tested on structural generalization—tasks requiring them to handle novel syntactic depths or center-embedded clauses—their performance often drops to near zero. As Bingzhi Li and others found with the SLOG benchmark, even significant increases in model depth do not resolve this fundamental difficulty in processing hierarchical nesting SLOG: A Structural Generalization Benchmark for Semantic Parsing.

The Search for Hierarchical Inductive Biases

Why do standard models fail? The consensus is a lack of hierarchical inductive bias. Standard Transformers use a "linear" attention mechanism that treats sequences as flat strings. To address this, researchers are exploring architectures that explicitly model tree structures:

Why It Matters: Beyond Shallow Parsing

The inability to handle recursion isn't just a linguistic curiosity; it limits the reasoning capabilities of Large Language Models (LLMs). Yuanhe Tian observed that while LLMs can generate long texts, the structural depth of their outputs often plateaus, failing to capture the true recursive complexity of human thought Large Language Models Are No Longer Shallow Parsers.

Furthermore, tasks involving metalinguistic self-reference (e.g., "This sentence has five words") remain a major hurdle. Jared Moore and Douwe Kiela found that even GPT-4 performs poorly on these tasks, suggesting that current architectures lack the recursive logic necessary to process language that describes its own structure I am a Strange Dataset: Metalinguistic Tests for Language Models.

By moving toward models that internalize hierarchical rules rather than memorizing linear patterns, the field aims to create AI that can reason through the infinite complexity of recursive systems.

Papers in this version

Showing 60 of 138 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 118 papers
  2. Version 1 · 2026-10-04 · 138 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_recursive_generalization_5c71b344,
  title = {recursive generalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/recursive-generalization/rdr_5c71b344c58c40a69983b80ee037b044},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_structural_generalization_for_deeply_nes_5c71b344_v1,
  title = {Structural Generalization for Deeply Nested Sequences},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/structural-generalization-for-deeply-nested-sequences/rdr_5c71b344c58c40a69983b80ee037b044/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}