Adapters as Data: The Latent Space of LoRAs

Moritz Grävinghoff, Konstantin Wehmeyer, Aron Asefaw, Boris Knyazev, Damian BortharXiv 2026 · arXiv

This paper, presented at the NeurIPS 2026 Workshop on Neural Network Artifacts, explores a fascinating shift in how we view machine learning models. Instead of treating Low-Rank Adaptation (LoRA) adapters as static, finished files, the authors propose treating them as a data modality—much like images or text—that can be mapped into a structured, navigable latent space.

TL;DR

The authors train a transformer autoencoder to learn a nonlinear latent space for LoRA adapters, moving beyond simple linear methods like PCA. This learned space not only preserves the ability to sample and edit identities but is significantly more robust to noise and can uniquely host adapters of different ranks within a single shared representation.

What Problem Does This Paper Address?

LoRAs have become the standard for fine-tuning large models (like Stable Diffusion) on specific tasks or identities. However, we usually treat each LoRA as an isolated artifact. Recent work, such as weights2weights (w2w), showed that these adapters actually live on a low-dimensional manifold that can be organized using linear Principal Component Analysis (PCA).

But linear methods have limits. They are fragile—small perturbations in weight space often cause the model to collapse into generating meaningless noise. Furthermore, weight-space methods are typically "rank-locked"; a system designed for rank-1 LoRAs cannot handle rank-2 LoRAs because the dimensionality is different. The authors ask: Can a learned, nonlinear latent space provide a more robust and flexible way to handle populations of adapters?

Methods

The authors adopt the SANE (Scalable Autoencoders for Neural network weights) methodology.

  1. Tokenization: They take the LoRA matrices ($A$ and $B$) and reshape them into a sequence of fixed-size tokens.
  2. Architecture: They use a GPT-style, non-causal transformer encoder-decoder. The encoder $E$ maps the weight sequence into a latent $z$, and the decoder $D$ reconstructs the weights.
  3. Self-Supervised Learning: The model is trained purely on a masked reconstruction objective. It doesn't "know" about faces or attributes during training; it only learns to compress and reconstruct weight tensors.
  4. Operations: Once the latent space is learned, they perform operations like:
  5. Sampling: Drawing new latent codes to generate entirely new identities.
  6. Editing: Using ridge regression to find "directions" in the latent space that correspond to attributes like "Male" or "Eyeglasses."
    • Perturbation: Adding noise to see how gracefully the representation degrades.
Overview of the approach, showing tokenization, the autoencoder, and various latent operations.

Datasets

  • Primary Dataset: A collection of ~65,000 rank-1 LoRA adapters (originally released by the w2w authors). These adapters were fine-tuned on Realistic Vision v5.1 (a Stable Diffusion 1.5 variant) to represent specific synthetic human identities.
  • Attributes: The adapters are labeled with 40 binary CelebA attributes (e.g., Smiling, Young, Gray Hair).
  • Mixed-Rank Data: The authors trained their own smaller pool of rank-2 adapters to test the model's ability to generalize across different architectures.

Key Insights and Ideas

  • The "Graceful Degradation" of Latents: This is perhaps the most striking finding. In raw weight space, adding noise quickly destroys the model's ability to generate a face, resulting in static-like textures. In the learned latent space, the same amount of noise results in a "valid" but different or slightly distorted face. The latent space "understands" the manifold of valid weights better than the raw numbers do.
  • Preservation of Linearity: Even though the autoencoder is nonlinear, the resulting latent space remains highly organized. Linear operations like PCA sampling and ridge-regression editing work just as well here as they do in raw weight space.
  • Breaking the Rank Barrier: By using a "sparse tokenizer" with padding, the authors successfully trained a single autoencoder that can process both rank-1 and rank-2 adapters. This suggests a path toward a "universal" latent space for adapters of varying architectures.

Key Results

  • Robustness: As shown in Figure 5, at a noise level of $\alpha=0.75$, weight-space perturbations have already collapsed into noise, while latent-space perturbations still yield recognizable (if varied) identities.
  • Identity Sampling: While "naive" sampling (treating every dimension as independent) leads to an "average" face, PCA sampling in the latent space produces high-quality, diverse identities (Figure 2).
  • Attribute Steering: The authors successfully found directions for rare attributes. For example, they could smoothly transition an identity to be "Bald" by moving along a specific vector in the latent space (Figure 3).
Comparison of noise robustness between weight-space and latent-space.

Limitations and Future Work

  • Reconstruction Floor: There is a slight loss in image fidelity just by passing an adapter through the autoencoder (an "LPIPS cost"). The authors note that every operation is observed through this "reconstruction floor."
  • Data Splitting: The authors acknowledge that their train/test splits were done per-adapter rather than per-identity, which might make some quantitative results (like AUC for attribute editing) slightly optimistic.
  • Domain Specificity: The study focused entirely on human faces. It remains to be seen if these latent spaces organize as well for non-human domains (e.g., artistic styles or different object classes).
  • Heterogeneity: While they bridged rank-1 and rank-2, the ultimate goal is a latent space that can handle entirely different adapter types or even different base models.

Summary updated

Abstract

LoRA adapters are usually treated as finished artifacts rather than as a data modality. Prior work has shown that LoRA adapters populate a low-dimensional manifold that a linear PCA basis already organizes well enough to support different operations such as sampling, editing, and inversion. We ask what a learned, nonlinear latent adds beyond this. We train a transformer autoencoder on the same population of LoRA identity adapters, mapping each into a latent space and compare every operation against the identical operation applied directly to the raw adapter weights. The latent preserves the capabilities already available in weight space, sampling diverse identities and editing individual attributes. Beyond preservation, we show that the latent is far more robust to displacement than weight space, degrading into distorted but still valid faces rather than texture. Fixed-dimensional weight-space representations cannot natively host adapters of more than one rank, so we extend our sequential autoencoder to hold mixed-rank adapters within one shared latent. This gives a path toward treating adapter weights of varying architecture as a single, general modality rather than one tied to a fixed rank, and points to a learned, nonlinear latent as the mechanism that could carry this generalization further, toward heterogeneous populations of adapters.
BibTeX
@misc{gravinghoff2026adapters,
  author = {Moritz Grävinghoff and Konstantin Wehmeyer and Aron Asefaw and Boris Knyazev and Damian Borth},
  title = {{Adapters as Data: The Latent Space of LoRAs}},
  year = {2026},
  eprint = {2611.07775},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07775},
}
Figure