Tracing Knowledge and Reasoning in LLMs

Yang Wang, Xiao Wang, Yiqi Liu, Chenghao Xiao, Eleni Vasilaki, Goran Nenadic, Chenghua Lin

Abstract

Current evaluations of Large Language Models (LLMs) report performance at the dataset or task level, revealing what a model achieves but offering little insight into the internal processing associated with its performance. We propose a task-centric framework for characterising the model-internal processing elicited by a benchmark. It combines attention- and activation-based signals with representation-based measures to characterise benchmarks spanning a continuum from retrieval-dominant to reasoning-dominant and to surface failure modes that task-level scores conceal. We evaluate 16 LLMs across 24 diverse benchmarks and conduct targeted internal-state analyses on representative models and tasks, observing systematic differences in processing between knowledge-centric and reasoning-centric tasks. We shift evaluation from asking what score a model achieves toward characterising the processing patterns associated with that score, potentially supporting more targeted benchmark selection and more informed model choice. We open source our code at https://github.com/extraordinarylab/trace-kr.
BibTeX
@misc{wang2026tracing,
  author = {Yang Wang and Xiao Wang and Yiqi Liu and Chenghao Xiao and Eleni Vasilaki and Goran Nenadic and Chenghua Lin},
  title = {{Tracing Knowledge and Reasoning in LLMs}},
  year = {2026},
  eprint = {2611.17461},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17461},
}
Figure