When a chatbot is shrunk to run cheaply, by storing its weights in 2 to 4 bits or by replacing weight matrices with low-rank approximations, the people who compress it check that it still answers trivia and math questions. They do not check whether it is still the same character. For role-playing agents, branded assistants, and safety personas, the character is the product. This proposal treats compression the way caricature research treats artistic exaggeration: a transformation that must not cross the point where the subject stops being recognizable. Caricature systems add an explicit identity term to their objectives so that heavy warping cannot erase who the face belongs to. We propose to give compression the same kind of term. We measure how much of a model's persona survives quantization and low-rank compression, and we steer the compressor's limited precision budget toward the weights that separate "this character" from "a generic assistant," instead of toward the weights that best predict text in general.
The core claim is that general-purpose and task-calibrated compressors protect the wrong thing for persona preservation. Likelihood of persona-flavored text is dominated by ordinary language tokens, so a compressor that preserves it can keep the model fluent and still let it slide into a sanitized assistant voice. A contrastive identity signal is different: it measures the gap in log-probability between an in-character response and a generic response to the same prompt, and it isolates exactly the weights whose damage causes persona drift. At matched bits per weight, we expect this signal to preserve character consistency where existing methods do not, at little or no cost in perplexity or MMLU.
What the two literatures give us
The vision side supplies the design principle rather than a module to reuse. WarpGAN trains its discriminator as a multi-class identity classifier, so any deformation that makes the person unidentifiable is penalized. 3D-CariGAN pairs an exaggeration loss with a "character loss" that keeps the identity direction collinear with the source. In both cases identity is enforced through a discriminative signal, meaning "is this still person A rather than someone else," not through pixel reconstruction. ACFun shows the cost of getting this wrong in the other direction: when identity constraints are too weak, style leaks in and overwrites facial structure. WebCaricature gives the evaluation lesson. Matchers trained on ordinary photographs fail badly on caricatures, so the evaluator itself must be matched to the identity question. These papers motivate the proposal. We borrow the discriminative identity objective and the warning about blind evaluators, not any image architecture.
The language side supplies the evidence for the gap and the machinery. Steering at the Source finds that persona traits in Qwen2.5-7B-Instruct are controlled by roughly three attention heads at one source layer. It also finds that MMLU is "blind" to the coherency collapse that strong steering causes, and that this localization fails in Gemma-3-12B, where persona directions are spread across layers. Chat Vector shows that chat style and safety behavior can be carried as a dense weight difference, which is a second hint that persona is not always confined to a few heads. Persona-Aware Contrastive Learning supplies the contrast we reuse: in-character responses conditioned on a full profile, against generic responses conditioned on only a name. It also adds a five-round "Chain of Persona" reasoning step that multiplies the tokens generated per turn, which is exactly the inference cost compression tries to remove. On the compression side, TACQ keeps the most salient weights in 16-bit precision and pushes the rest to 2 to 3 bits, scoring saliency by simulated quantization error and by magnitude-sharpened gradients of a task loss. ASVD and AIR do activation-aware and influence-aware low-rank truncation. All three evaluate only perplexity and reasoning benchmarks. We reuse their machinery and change only the signal that decides what to protect.
Core mechanism
Fix a character profile $P$, a dialogue context $x$, an in-character reply $y^+$ produced by the uncompressed model given $P$, and a generic reply $y^-$ produced given only the character's name. Define the identity margin of weights $\theta$ as
$$m_\theta(x) = \log p_\theta(y^+ \mid x, P) - \log p_\theta(y^- \mid x, P),$$
which is large when the model clearly prefers its character's voice over the generic assistant's. This plays the role of WarpGAN's identity classifier: it scores identity discriminatively rather than by reconstruction. Compression then preserves the sum of the general language-modeling loss and $\lambda$ times the drop in margin relative to the uncompressed model, where $\lambda$ is the single trade-off weight. Per-weight saliency follows TACQ's quantization-aware estimate. In place of TACQ's task-loss gradient we use the gradient of this combined objective. For low-rank compression, the same combined gradient becomes the influence weight in AIR's alternating-least-squares sweep. Both instantiations share the calibration budget and the backward-pass cost of their parent methods. We expect $\lambda$ to trace a frontier between persona retention and perplexity, the language-model counterpart of the exaggeration-versus-identity curve in caricature work.
Research Program
The first stage is a measurement study, and the rest of the project depends on its outcome. We compress Qwen2.5-7B-Instruct, where persona heads are localized; Llama-3-8B-Instruct, where TACQ reports results; and Gemma-3-12B, where persona is distributed. Each is compressed with GPTQ, TACQ, ASVD, and AIR at 4, 3.1, and about 2.5 average bits, and at 80% and 60% parameter retention. We then measure persona retention alongside the usual metrics. If persona degrades no faster than MMLU, the problem is not real and we stop with a negative-result paper, which would still be useful given that no one has tested it.
The second stage asks where the damage lands. We compare the per-head and per-layer saliency maps produced by the identity margin with the Head Contribution Scores from Steering at the Source. This tells us whether role-play consistency against a profile lives in the same few heads as trait steering or is spread out the way Chat Vector suggests.
The third stage builds and evaluates identity-margin TACQ and identity-margin AIR. It also tests a deployment question: can a 3-bit identity-protected model without Chain of Persona match a 16-bit model with it on character consistency, at a fraction of the per-turn latency and KV-cache growth?
Baselines
The naive reference is the uncompressed 16-bit model with the profile in its system prompt. It defines 100% persona retention and the latency we aim to cut. A second reference is plain GPTQ at each bit width with generic calibration text. This shows the damage a default deployment would incur.
The strongest existing competitor for quantization is TACQ run with persona dialogues as its calibration data and the likelihood of in-character replies as its task loss. It is the strongest competitor because it already does task-conditioned, mixed-precision protection, and pointing it at persona data is the obvious move a reviewer would make. On Llama-3-8B-Instruct, TACQ reports recovering 96% of unquantized MMLU at 3.1 bits, a 5.25-point absolute gain over SpQR, and an average 14.74% gain over SliM-LLM at 2 bits. No reported result was found on Lacuna for TACQ or any compressor on CharacterEval or persona-trait metrics. We will establish those numbers ourselves.
For low-rank compression, the strongest competitor is AIR with general-data influence. AIR reports more than 18% better perplexity than SVD-LLM(W) at 60% or less parameter retention, and a 53% per-token latency reduction at 60% retention.
Fairness is enforced by holding fixed the calibration dialogues (identical prompts and token counts), the average bits per weight including the 16-bit outliers, the rank budget, the number of backward passes, and a shared grid of hyperparameter settings per method.
The key ablation swaps the contrastive margin for plain likelihood of $y^+$ with everything else identical. This isolates whether the discriminative identity signal, not just persona-domain data, is responsible for any gain.
Beating the baselines means two things at matched bits. First, a statistically significant gain in CharacterEval character consistency (scored by CharacterRM) and in the trait-versus-coherency envelope area, across at least two of the three models. Second, perplexity and MMLU within one point of TACQ.
Why not just…?
Why not just run TACQ with persona data as calibration? This is the most serious threat. If it matches us, the contribution reduces to "calibrate on the right data," and we will say so. Our reason to expect a difference is that likelihood of an in-character reply is mostly driven by tokens any fluent model predicts well, so its gradient is dominated by general language weights. The margin cancels that shared component. The likelihood-versus-margin ablation above answers this question directly. We will report it even if the margin loses.
Why not just keep the three style heads in 16-bit precision? That costs almost nothing, and on Qwen2.5 it might suffice for trait steering. But the localization reported in Steering at the Source breaks down on Gemma-3-12B. Profile-level role-play consistency, which is what CharacterEval measures, may not live in heads found for single traits like "Humorous." We include "protect the HCS heads at matched bit budget" as an explicit baseline. If it ties our method on Qwen but fails on Gemma, that is the expected and informative pattern.
Why not just compress normally and restore the persona afterward with a longer prompt, Chain of Persona, or steering? Prompted personas already drift beyond ten turns on CharacterEval. Chain of Persona multiplies generated tokens, and global residual steering causes coherency collapse according to Steering at the Source. Each of these gives back some of the latency compression bought. We will measure the end-to-end cost directly by comparing 3-bit-plus-Chain-of-Persona against identity-protected 3-bit without it, on both consistency and wall-clock time per turn.
Why not just fine-tune the compressed model on role-play data? Recovery fine-tuning is a legitimate alternative, but it needs per-character training and risks forgetting, which is the concern Persona-Aware Contrastive Learning addresses by checking ARC and TriviaQA. Our method is post-training and uses only forward and backward passes on a small calibration set. We will report a LoRA-recovery arm at equal GPU-hours as an upper reference rather than as a competitor to beat.
Proposed Experiments
Persona evaluation uses three instruments. The first is CharacterEval's 77 profiles and multi-turn dialogues, scored by CharacterEval's CharacterRM, which correlates with humans at 0.631 Pearson versus 0.385 for GPT-4. We report results separately for turns beyond ten, where drift concentrates. The second is the six-trait persona suite with the Right-normalized Constrained Envelope Area from Steering at the Source. The third is IFEval. General capability uses WikiText-2 perplexity and MMLU. Cost uses average bits per weight, peak memory, and per-token latency.
Calibration profiles and evaluation profiles are disjoint, so we test whether protection generalizes across characters rather than memorizing one. To guard against the evaluator problem WebCaricature exposed, we run a small human preference study (in-character or not) on a sample of 4-bit and 2.5-bit outputs and check its agreement with CharacterRM before trusting automated scores at extreme compression.
The decisive checks are, in order, the following. Does persona retention fall faster than MMLU retention under standard compression? This is the go/no-go question. Does the margin beat likelihood-calibrated TACQ at matched bits? Does any gain survive on Gemma-3-12B, where head-level protection should fail? The project succeeds if the answer to all three is yes. It still yields a clear, publishable map of where compression erases character if only the first is.
References
- WarpGAN: Automatic Caricature Generation.
- 3D-CariGAN: An End-to-End Solution to 3D Caricature Generation From Normal Face Photos
- ACFun: Abstract-Concrete Fusion Facial Stylization
- WebCaricature: a benchmark for caricature recognition
- Steering at the Source: Style Modulation Heads for Robust Persona Control
- Chat Vector: A Simple Approach to Equip LLMs With New Language Chat Capabilities
- Enhancing Persona Consistency for LLMs' Role-Playing using Persona-Aware Contrastive Learning
- Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression
- ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models
- Activation- and Influence-Aware Ranks (AIR): Function-Preserving SVD Compression for LLMs
- CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation