TRACE for Tracking the Emergence of Semantic Representations in Transformers

Nura Aljaafari, Danilo S. Carvalho, André FreitasarXiv 2025 · DOI · arXiv

Abstract

Understanding the internal learning dynamics of Transformers remains a significant challenge in AI research. This paper introduces TRACE (Tracking Representation Abstraction and Compositional Emergence), a diagnostic framework designed to pinpoint the "phase transition" where models shift from rote memorization to abstract linguistic understanding. By monitoring the intersection of loss landscape curvature and the intrinsic dimensionality of hidden states, we demonstrate that models undergo a distinct representational shift that correlates with functional mastery of semantic roles.

Introduction

While Transformers have revolutionized natural language processing, the "black box" nature of their training obscures when and how they develop structural understanding. Most research focuses on end-state convergence or simple symbolic tasks. TRACE addresses this gap by tracking the emergence of holistic, sentence-level semantic representations in real-time. Identifying these transitions is crucial for improving training efficiency and model steering.

The TRACE Framework

The authors propose a multi-signal diagnostic framework that monitors three distinct data types during training:

  1. Geometric Signals: A Curvature Complexity Score $C(H)$ derived from the Hessian of the loss function. High curvature indicates active reorganization or memorization, while a "collapse" into flatter regions suggests the discovery of generalizable abstractions.
  2. Statistical Signals: Intrinsic Dimensionality (ID) measures the minimum degrees of freedom in hidden states. ID typically rises as features are learned and stabilizes as they are compressed into abstract categories.
  3. Linguistic Signals: Internal Probing and Output Generation Analysis verify if internal representations align with linguistic predictions.
Overview of the TRACE framework, illustrating the integration of geometric, statistical, and linguistic monitors during Transformer training.

Experimental Setup: ABSynth

To isolate learning dynamics from the noise of natural language, the researchers developed ABSynth, a synthetic data generator grounded in frame semantics. Unlike template-based systems, ABSynth samples from abstract "event frames" (e.g., a "Motion" event requiring a Source and a Goal). The resulting ABSynth25K dataset provides perfect ground-truth labels for semantic roles, allowing precise tracking of what the model learns at every step.

Key Insights: The "Aha!" Moment

The study reveals several critical aspects of the learning process:

  • Geometric Signature of Abstraction: The onset of abstraction is marked by a consistent intersection between falling curvature and stabilizing intrinsic dimensionality. This intersection serves as a reliable marker for a representational phase shift.
  • The Two-Phase Dissociation: Models often achieve high surface-level accuracy early on, but their internal representations continue to reorganize for thousands of steps before reaching a stable abstract state.
  • FFNs as Stabilizers: Ablation studies show that while models can reach abstraction without Feed-Forward Networks (FFNs), the process becomes significantly more volatile. FFNs appear to act as distributed lookup tables that smooth the learning trajectory.

Key Results

Across various model sizes and configurations, a robust two-phase dynamic was observed: an initial regime of rising ID and high curvature, followed by a transition into flatter curvature and stable ID.

Curvature and Intrinsic Dimension across different model sizes and architectural configurations (Standard, Without FNN, and Reduced Attention Heads).

Larger models exhibit sharper and earlier "aha!" moments. After this identified phase transition, even small models achieved near-perfect token accuracy (98%) and high exact match scores (0.84) on synthetic tasks, confirming that these geometric markers are reliable indicators of functional mastery.

Conclusion and Future Work

TRACE provides a powerful lens into the "black box" of Transformer training. Future work will focus on applying these signatures to large-scale natural language pre-training and using them as early-stopping criteria to optimize computational resources. Integrating TRACE with mechanistic interpretability tools like Sparse Autoencoders could further pinpoint the specific circuits formed during these critical phase transitions.

Summary updated

BibTeX
@misc{aljaafari2025trace,
  author = {Nura Aljaafari and Danilo S. Carvalho and André Freitas},
  title = {{TRACE for Tracking the Emergence of Semantic Representations in Transformers}},
  year = {2025},
  eprint = {2505.17998},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.17998},
}
Figure