Introduction
While the field has achieved remarkable success in training Large Language Models (LLMs), our understanding of the internal mechanisms by which these models acquire language remains limited. Standard training metrics, such as loss curves or gradient norms, often mask complex developmental milestones. A model may exhibit a smooth decrease in loss while its internal representations undergo significant structural reorganizations, or it may achieve high accuracy through memorization rather than genuine syntactic understanding. This "blind spot" complicates decisions regarding training duration, architectural comparisons, and structural debugging.
To address this, we present TRACE (Tracking Representation Abstraction and Compositional Emergence), a modular toolkit designed to monitor a transformer model's internal state in real-time. TRACE moves beyond post-hoc analysis, providing researchers with live insights into how a model's "brain" evolves during the training process.
Methodology: The TRACE Framework
TRACE is designed as a "plug-and-play" framework for PyTorch training loops, consisting of three primary pillars: monitoring hooks, analysis modules, and output diagnostics.
The framework utilizes lightweight hooks to capture hidden states and gradients with minimal computational overhead. These captured signals are processed by several analysis modules:
- Linguistic Probes: These modules evaluate internal layers' ability to encode specific linguistic features, such as parts of speech (POS) or semantic roles (e.g., Agent, Patient).
- Intrinsic Dimensionality (ID): This geometric metric assesses the complexity and compression of internal representations, signaling shifts from memorization to abstraction.
- Hessian Analysis: By examining the curvature of the loss landscape, TRACE identifies volatile learning phases versus periods of stability and generalization.
Experimental Framework
To validate TRACE, we developed ABSynth, a controllable synthetic corpus generator. Unlike natural language, where features are often entangled, ABSynth allows for the generation of data with precise, "gold-standard" annotations for specific linguistic structures. For our demonstration, a 2-layer decoder-only transformer was trained on 50,000 synthetic sentences following a Zipfian distribution, incorporating structured metadata for semantic roles.
Results and Discussion: Mapping the Evolution of Learning
Our analysis using TRACE revealed several critical insights into the learning dynamics of transformer models that are invisible to standard metrics.
The Stability Gap and Learning Hierarchy
A key finding is the "stability gap": a model's external performance often plateaus long before its internal representations stabilize. Furthermore, models exhibit a clear hierarchy in feature acquisition. Core semantic roles, such as Agents and Actions, emerge and stabilize early in training. In contrast, "adjunct" roles like Locations are acquired much later and exhibit higher volatility.
Geometric Signals of Abstraction
Geometric analysis provided evidence for a "memorization-to-abstraction" phase shift. We observed an initial drop in intrinsic dimensionality as the model simplified the input data, followed by a rebound as it built more complex, generalized internal structures. Additionally, spikes in Hessian curvature—indicating a "bumpy" loss landscape—frequently served as early warning signals for major structural shifts in how the model represents data.
Conclusion and Future Directions
TRACE provides a powerful lens for observing the non-linear path of machine learning. By identifying developmental phases and linguistic emergence in real-time, it offers actionable diagnostics for model training. Future work will focus on scaling TRACE to natural language datasets through integration with tools like spaCy for automated annotation and expanding compatibility with large-scale models via the Hugging Face ecosystem.
Summary updated