TL;DR
EDITORS (Editing LoRA Subspaces) is a novel "diagnose-then-deflate" framework that disentangles writing style from semantic content in Large Language Models (LLMs). By identifying and suppressing content-oriented directions in the model's activation space during fine-tuning, it achieves state-of-the-art performance in authorship attribution and generates stylistically faithful text without simply memorizing training data.
What Problem Does This Paper Address?
In the world of literary text modeling, style and content are deeply entangled. This creates two major hurdles for current LLMs:
- Authorship Attribution (AA): Models often rely on "content shortcuts." If an author frequently writes about politics, the model might identify them based on the topic rather than their unique syntax or vocabulary. This leads to failure when the author switches topics or domains.
- Style Imitation (SI): When asked to write in a specific style, models tend to "hallucinate" or copy-paste specific facts from their training data (memorization) rather than adopting the author's abstract stylistic patterns.
Existing solutions often force a hard trade-off: they either retain too much content (leading to shortcut learning) or remove content so aggressively that the resulting text loses its meaning or discriminative power.
Methods
The authors propose EDITORS, which uses a two-stage geometric intervention within the LLM's representation space:
- Diagnosis: The framework first extracts "style-neutral statements" (atomic facts) from the training corpus using an external LLM. It trains a diagnostic LoRA (Low-Rank Adaptation) adapter on these facts. Because this adapter is trained only on neutral content, its weights represent the directions in the activation space most associated with what is being said.
- Deflation: The authors apply Singular Value Decomposition (SVD) to these diagnostic weights to identify a "content-oriented subspace."
- Style Learning: A final style adapter is trained on the original author-labeled text. Crucially, EDITORS uses an activation-space deflation hook. During training, input activations are projected away from the identified content subspace. This forces the style adapter to learn from the "residual" signal—the stylistic features that remain after the dominant content signals are suppressed.
During inference, the frozen base model provides the necessary semantic support, while the style adapter injects the specific authorial "flavor."
Datasets
The researchers evaluated EDITORS across three distinct domains:
- CCAT10/50: A news-based benchmark (Reuters) focusing on corporate and industrial topics.
- Blogs10/50: A social media dataset containing informal, noisy, and imbalanced writing.
- Gutenberg30: A long-form literary benchmark derived from Project Gutenberg, featuring 30 authors with distinct voices. This was the primary dataset for Style Imitation tasks.
Key Insights and Ideas
- Partial Separability: The core intuition is that while style and content are entangled, they are partially separable in the high-dimensional activation space of an LLM.
- Activation vs. Parameter Intervention: Unlike previous methods that try to constrain the weights of a model, EDITORS intervenes on the activations. This is a more direct way to control what information the model "sees" during the forward pass, preventing semantic leakage through the frozen backbone.
- The "Warm-up" Duality: For Authorship Attribution, a brief "domain warm-up" (training on the target domain before deflation) helps accuracy. However, for Style Imitation, this same warm-up is detrimental because it encourages the model to memorize specific authorial topics.
- Scaling Benefits: The method shows consistent gains across various architectures (Llama, Mistral, Gemma, Phi). Interestingly, larger models often show higher gains, likely because they have more redundant capacity for the deflation process to exploit.
Key Results
EDITORS demonstrated significant improvements over strong baselines like BertAA, STAR, and standard SFT:
- Authorship Attribution: EDITORS achieved state-of-the-art accuracy, outperforming the strongest baselines by up to 29% on challenging 50-author tasks. As shown in the table below, it consistently beat standard Llama-3.1 SFT across all benchmarks.
- Style Imitation: In SI tasks, EDITORS achieved higher style similarity scores than GPT-4 and StyleTunedLM while significantly reducing memorization (measured by lower BLEU and Self-BLEU scores). This indicates the model is truly imitating the style rather than just repeating the training text.
Limitations and Future Work
- Evaluation Bottlenecks: The study relies heavily on "LLM-as-a-judge" and ensembles of classifiers for style evaluation. While these are currently the best automated tools available, they may not fully capture the nuances that a human literary expert would notice.
- Dataset Constraints: There is a lack of high-quality "parallel" datasets where the exact same content is written by multiple authors in their respective styles. Creating such benchmarks would allow for even more rigorous testing of style-content disentanglement.
- Ethical Risks: The authors acknowledge that high-performance AA and SI could be misused for deanonymization or creating sophisticated deepfake text. They advocate for the use of watermarking and other safeguards in practical deployments.
Summary updated