TextNCA: Neural Cellular Automata for Language Modeling via Hierarchical Local Attention

Avni Mittal, Avinash Anand, Ashutosh Kumar, Dikshant Kukreja, Kritarth Prasad, Sushane Dulloo, Erik Cambria, Timothy Liu, Zhengkui Wang, Rajiv Ratn ShaharXiv 2026 · arXiv

TL;DR

TextNCA is a new analytical probe that applies the principles of Neural Cellular Automata (NCA)—local perception, weight-shared iteration, and gated updates—to language modeling. The researchers discovered that the model's performance is primarily driven by a hierarchical narrow-to-wide attention schedule rather than the iterative weight sharing itself, revealing a "prepare-then-decode" mechanism in the model's internal logic.

What Problem Does This Paper Address?

Modern Large Language Models (LLMs) are dominated by the Transformer architecture, which relies on global self-attention and a stack of distinct, non-shared layers. In contrast, Neural Cellular Automata (NCAs) compute by repeatedly applying a single, shared local rule. While NCAs have been highly successful in image generation and texture synthesis, their utility in the discrete, sequential world of natural language remains largely unexplored.

The authors ask a fundamental question: Can the NCA primitive support language modeling, and if so, which of its core properties—locality, iteration, or weight sharing—actually drives the model's behavior?

Methods

The authors introduce TextNCA, a 1D causal windowed-attention realization of the NCA primitive. To make it work for language, they utilize a hierarchical staged architecture:

  1. Core Update: Each step uses local causal softmax attention within a window $w$, followed by a LayerNorm, an FFN, and a GRU-style gate to update the token state in place.
  2. Staging: Instead of a single loop, the model cascades three stages. Each stage has its own set of weights but iterates them $T_s$ times.
  3. Window Schedule: The stages follow a "narrow-to-wide" progression with window sizes $w \in \{8, 32, 128\}$.
  4. Step Embeddings: To allow the shared weights to specialize across iterations, a unique learned embedding is added at each step.
The Hierarchical TextNCA architecture showing the staged pipeline and the effective receptive field expansion.

Datasets

  • Pre-training: A 500M-token subset of SlimPajama, tokenized with the Mistral 32K BPE vocabulary.
  • Language Modeling Eval: WikiText-103 (measuring token-level perplexity).
  • Downstream Tasks: IMDb and SST-2 (sentiment analysis), AG News (topic classification), and SQuAD v1.1 (extractive QA).

Key Insights and Ideas

  • The Schedule is the "Load-Bearing" Wall: The most critical design choice is the narrow-to-wide window schedule. Reversing this schedule (starting wide and going narrow) causes a massive +70.8 PPL (perplexity) penalty. In contrast, removing iteration and weight sharing entirely while keeping the schedule (a model they call SWin-TF-Staged) only results in a minor +4.1 PPL increase.
  • Prepare, then Decode: By probing the hidden states, the authors found a distinct functional split. Stages 1 and 2 (narrower windows) focus on "representation preparation"—organizing topic structure and syntax—without actually reducing next-token loss. The final stage (wide window) performs the actual "decoding" into next-token probabilities.
  • Iteration as Effective Depth: Iteration in TextNCA doesn't act like a "refinement" process that converges to a fixed point. Instead, it behaves like a depth knob. There is a clear optimum ($T_s=4$); adding too many iterations actually degrades performance as the hidden states drift away from what the language model head expects.
  • Non-Monotone Quality: Interestingly, linear probing accuracy for topic classification peaks in the middle stage ($w=32$) before dropping slightly in the final stage as the model trades general representation quality for specific next-token readiness.
Figure 3: Per-step linear probing accuracy across the 12 NCA steps (embedding shown as step 0). A 5-way topic classifier trained on hidden states at each step shows that representation quality is non-monotone: accuracy jumps from 56.7% at the embedding to 77.5% after the w=8 stage, peaks at 81.7% during w=32, then drops to 79.2% through w=128.

Key Results

  • Performance Gap: At a 30M parameter scale, Hier-TextNCA reaches 60.3 PPL, trailing behind a standard 6-layer Transformer (52.8 PPL) and a 12-layer Transformer (44.7 PPL).
  • Ablation Extremes: Reversing the window schedule leads to a disastrous 131.1 PPL, while a uniform window size results in 119.7 PPL.
  • Downstream Transfer: TextNCA is statistically tied with Transformers on classification tasks (averaging ~91.2% accuracy) but struggles significantly on SQuAD (F1 of 26.3 vs 33.2 for a 12L Transformer), likely due to the inherent limitations of local attention for span-level retrieval.
Comparison of PPL across different schedule perturbations and controls.

Limitations and Future Work

  • Computational Efficiency: TextNCA is currently not Pareto-superior to Transformers. It requires ~1.9x more FLOPs and has ~2.7x lower throughput than a matched Transformer because the sequential shared updates cannot be fused.
  • Locality Bottleneck: The strict local attention limits performance on tasks requiring global context access, such as complex question answering.
  • Scaling: The study was conducted at the 30M parameter scale; whether iteration becomes a more significant contributor at the billion-parameter scale remains an open question.
  • Future Directions: The authors suggest exploring more sophisticated per-step conditioning, such as Adaptive Computation Time (ACT) or FiLM-style modulation, to better harness the power of iteration.

Summary updated

Abstract

Can a strictly local, iterated, weight-shared computation primitive support language modelling, and which of those three properties actually drives the model's behaviour? We define \textsc{TextNCA}, a 1D causal windowed-attention realisation of the Neural Cellular Automaton primitive, and study a hierarchical variant that cascades three stages with windows $w \in \{8, 32, 128\}$ and $T_s$ shared-weight iterations per stage, all on WikiText-103 at roughly 30M parameters and 60k training steps. The model does not match a parameter-matched Transformer at this scale (Hier-TextNCA $60.3$ vs.\ Transformer-6L $52.8$ and Transformer-12L $44.7$ PPL), so we treat it as an analytical probe rather than a proposed alternative. The behaviour we observe is largely explained by the staged narrow-to-wide schedule: a non-iterating sliding-window Transformer that reuses the same schedule comes within $+4.1$ PPL of the iterated model, while reversing, flattening, or breaking the monotonic ordering of the schedule costs between $+16.7$ and $+70.8$ PPL. Iteration adds a smaller bounded benefit on top of the schedule, with a clear optimum at $T_s{=}4$ and a U-shaped degradation beyond it. The GRU gate and learned per-step embeddings are required for that benefit to appear, and training with random $T_s$ yields an inference-time iteration-count knob at the cost of substantially higher absolute PPL. We position the work as a controlled reading of which parts of NCA-style computation carry the weight in language modelling.
BibTeX
@misc{mittal2026textnca,
  author = {Avni Mittal and Avinash Anand and Ashutosh Kumar and Dikshant Kukreja and Kritarth Prasad and Sushane Dulloo and Erik Cambria and Timothy Liu and Zhengkui Wang and Rajiv Ratn Shah},
  title = {{TextNCA: Neural Cellular Automata for Language Modeling via Hierarchical Local Attention}},
  year = {2026},
  eprint = {2608.02050},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.02050},
}
Figure