Abstract
While Large Language Models (LLMs) demonstrate remarkable fluency, the internal mechanisms by which they represent and control complex grammatical structures remain poorly understood. This study investigates the latent representation of verb tense and aspect in LLMs. By identifying orthogonal directions in the models' residual streams using Linear Discriminant Analysis (LDA), we demonstrate that it is possible to steer multi-token generation toward specific grammatical forms. Our findings reveal that while tense and aspect are represented in distinct subspaces, effective steering requires precise calibration of intervention strength and timing to mitigate unintended semantic drift.
1. Introduction
The ability of Large Language Models (LLMs) to generate grammatically correct text is well-documented, yet the internal encoding of syntactic knowledge is still a subject of intense research. Most prior work in mechanistic interpretability has focused on binary contrasts—such as singular versus plural—and evaluated model behavior on single-token continuations. However, natural language is inherently compositional and hierarchical, often requiring consistency across multiple tokens.
This paper addresses this gap by examining two multidimensional grammatical phenomena: tense (past, present, future) and aspect (simple, progressive, perfect, and perfect progressive). We explore whether these features can be localized within the model's latent space and if they can be causally manipulated to control generation without compromising the model's overall coherence.
2. Methodology
The research focuses on two state-of-the-art models: Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct. The methodology follows a three-stage pipeline: localization, geometric analysis, and causal intervention.
2.1 Localization and Geometry
To identify where grammatical features reside, we trained linear classifiers on aggregated hidden states across all layers. To move beyond simple binary classification, we employed a variant of Linear Discriminant Analysis (LDA). This allowed us to find normalized class directions for categorical features, effectively isolating distinct vectors in the residual space for specific tenses and aspects.
2.2 Steering Techniques
We implemented three primary steering methods to modify the residual stream during generation:
- Target-Addition (TA): Adding the target feature vector to the activation.
- Target-Addition + Source-Subtraction (TA+SS): Adding the target vector while simultaneously subtracting the vector corresponding to the original feature.
- Target-Addition + Projection Subtraction (TA+Proj-SS): Adding the target vector and subtracting only the component of the activation that aligns with the source direction.
These interventions were tested across three tasks: Random Sentence generation, Repetition (copying a provided sentence), and Temporal Translation (rewriting a sentence in a different tense or aspect).
3. Data and Evaluation
The study utilized a refined version of the Penn Treebank annotated with PropBank, filtered for single-verb constructions to ensure label clarity. To address class imbalances—particularly the rarity of perfect progressive forms—the dataset was augmented with synthetic examples generated by GPT-4o, resulting in a balanced set of 1,543 labeled sentences. Evaluation was further validated using a 281-sentence subset from the BIG-bench benchmark.
4. Key Findings
4.1 Orthogonality of Grammar
A significant discovery is that tense and aspect are encoded in near-orthogonal subspaces. This suggests that LLMs naturally learn a structured, independent representation of these concepts, mirroring the categorical distinctions made in formal linguistics.
4.2 The Dynamics of Steering
The efficacy of steering is highly dependent on the model's internal activation norms. We observed that activation norms grow significantly in deeper layers. Consequently, Qwen-2.5-7B-Instruct required much higher steering multipliers ($\alpha = 100-250$) compared to Llama-3.1-8B-Instruct ($\alpha = 5-25$).
Furthermore, the timing of the intervention is critical. Steering is most effective when applied during the generation of the tokens themselves, specifically right before the model generates the verb, rather than during the processing of the initial prompt.
4.3 Performance and Side Effects
Probing results were exceptionally high, with F1 scores exceeding 0.90 at the embedding layer and reaching near-perfection (1.0 for tense, 0.98 for aspect) in deeper layers. However, steering efficacy varied:
- Tense vs. Aspect: Tense proved significantly easier to steer (94-96% success) than aspect (51-66%).
- Task Resistance: Few-shot tasks like Repetition and Temporal Translation were more resistant to steering, as the contextual prompt often conflicted with the steering vector.
- Topic Shift: While steering can successfully change grammar, it often induces "topic shift" or semantic drift. Maintaining semantic consistency (measured via BERTScore) remains a challenge when applying strong steering vectors.
5. Conclusion and Future Work
This study demonstrates that LLMs possess highly structured internal representations of complex grammatical features like tense and aspect. While these features can be steered, the process is sensitive to intervention strength and task context.
Future research should investigate how these representations interact with temporal adverbs and explore automated methods for optimizing steering parameters. Additionally, expanding this work to morphologically rich or non-Indo-European languages will be essential to determine the universality of these latent structures.
Summary updated