Abstract
Language adapters have emerged as a highly effective, parameter-efficient solution for cross-lingual transfer in large language models. However, the internal mechanics of how these tiny modules transform a source-language model into a target-language one have remained largely opaque. This paper investigates the internal "hidden space" of these adapters using a suite of interpretability techniques, including the Logit Lens, iterative ablation, and geometric analysis. Our findings reveal that adapters do not reinvent linguistic representations; instead, they "piggyback" on the source language's internal logic. The transition to the target language is a "late-shift" phenomenon, occurring primarily in the final layers of the network. Moreover, adapters preserve the structural geometry of the original model, shifting representations toward the target language's embedding space without disrupting established linguistic clusters.
Introduction
As Transformer-based language models grow in scale, full fine-tuning for every new language becomes computationally prohibitive. Adapters—small, trainable modules inserted into a frozen pre-trained model—offer a solution by fine-tuning less than 1% of the total parameters. While empirically successful, the "how" and "where" of this adaptation are unknown. Do adapters create a separate "German" subspace? Do they translate layer-by-layer? Understanding these dynamics is essential for optimizing model architectures and improving cross-lingual transfer, especially for low-resource languages.
Methodology
To dissect the behavior of language adapters, we employed four primary analytical lenses:
- The Logit Lens: By projecting intermediate hidden states directly into the vocabulary space, we can observe the model's "top-of-mind" predictions at each layer.
- Iterative Ablation: We systematically disabled adapter modules at various depths to quantify their contribution to the final output perplexity.
- Sparse Probing: Linear classifiers were trained to detect if adapted features occupy a distinct, isolated subspace within the residual stream.
- Geometric Analysis (PCA): We compared the "shape" of the representation space (e.g., clustering of parts of speech) between the original and adapted models using Principal Component Analysis.
Experiments were conducted using Wikipedia for adaptation and evaluated on the FLORES-101 translation dataset and PUD Parallel Universal Dependencies for linguistic structure analysis.
The "Late-Language" Shift
A central discovery of this study is the "late-language" shift. For approximately 90% of the model's depth, intermediate representations remain rooted in the source language (English). As shown in Figure 4, target language tokens only begin to dominate the top predictions in the final two layers of a 24-layer model. This suggests that the bulk of the model's computation is language-agnostic or source-language-centric, with the adapters performing a final "translation" step just before the output.
Structural Preservation and Redundancy
Contrary to the hypothesis that adapters might reorganize the model's internal logic, our geometric analysis shows high alignment (~0.6 cosine similarity) between the principal components of the source and adapted spaces. Adapters appear to "shift" the entire cloud of representations toward the target language's embeddings while maintaining the relative distances between concepts (e.g., verb clusters remain verb clusters).
Furthermore, our ablation studies (Figure 5) indicate that the adaptation process is highly redundant in early and middle layers. Removing these modules often results in negligible performance degradation. However, the final layers are "load-bearing"; their removal causes a catastrophic spike in perplexity, confirming that the most critical language-specific transformations occur at the end of the inference pipeline.
Conclusion and Future Work
This research provides a new mental model for language adapters: they are steering mechanisms rather than re-inventors. The "late-shift" dynamic suggests significant opportunities for "Adapter Dropping"—pruning redundant early-layer adapters to accelerate inference. Future work should investigate whether these dynamics hold at the scale of "frontier" models and explore specialized architectures for linguistically distant languages where the "work" required by adapters is significantly higher.
Summary updated