Interpretability of Language Models for Learning Hierarchical Structures

Zeyuan Allen-Zhu, Yuanzhi LiOpenReview 2025 · Submission to ICLR 2025 · Rejected · OpenReview

Abstract

Transformer-based language models are effective but complex, and understanding their inner workings is a significant challenge. Previous research has primarily explored how these models handle simple tasks like name copying or selection, and we extend this by investigating how these models grasp complex, recursive language structures defined by context-free grammars (CFGs). We introduce a family of synthetic CFGs that produce hierarchical rules, capable of generating lengthy sentences (e.g., hundreds of tokens) that are locally ambiguous and require dynamic programming to parse. Despite this complexity, we demonstrate that generative models like GPT can accurately learn this CFG language and generate sentences based on it. We explore the model's internals, revealing that its hidden states precisely capture the structure of CFGs, and its attention patterns resemble the information passing in a dynamic programming algorithm.
BibTeX
@misc{allenzhu2025interpretability,
  author = {Zeyuan Allen-Zhu and Yuanzhi Li},
  title = {{Interpretability of Language Models for Learning Hierarchical Structures}},
  year = {2025},
  url = {https://openreview.net/forum?id=J6qrIjTzoM},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2025},
}
Figure