When people instruct a language model, they often ask for several things at once: "summarize this in about sixty words, make sure it mentions contrastive loss and ImageNet, and if you can't fit everything, keep it short." Current models handle such requests by reading the whole instruction as one block of text and hoping the result respects every part of it. That works reasonably well when the constraints agree. It breaks down when they conflict, when the user asks for a matter of degree ("a bit shorter"), or when the combination of constraints was never seen in training. This proposal asks whether a model can instead split an instruction into its separate requests and turn each one into a setting on its own small, separately trained module. Each module would get a strength, and the user's stated priority would decide how the strengths trade off.
The modules are low-rank adapters (LoRA): small add-on weight matrices, each trained for one skill while the large model stays frozen. The core claim is that per-attribute adapters, kept in non-overlapping weight directions and scaled by strengths predicted from each clause of a free-form instruction, give three things a single instruction-conditioned model does not. The effect changes steadily as the requested amount changes. The model follows a stated priority when constraints conflict. And quality holds up on constraint combinations not seen in training. In two sentences: language says which attribute and how much, and a small network turns that into a strength on a dedicated adapter. Because the adapters are kept from overlapping, their effects add roughly independently, so a priority stated in words becomes a ratio of strengths.
What exists, and what is missing
Several papers already turn text into adapter weights, so the obvious version of this idea is taken. Text-to-LoRA trains a hypernetwork (a network that outputs the weights of another network) to produce a full adapter from a task description in one forward pass. Its authors report that it is sensitive to phrasing and works only at the level of a whole task. SHINE compiles long contexts into adapter weights. LoRA-Gen reads a system prompt and picks and merges experts from a pool, and reports that predicting weights directly overfits. Semantic-guided LoRA Parameters Generation fuses experts by how similar their descriptions are to the request. All four map one instruction to one static adapter for one task. None of them breaks an instruction into separate constraints, sets a magnitude per constraint, or has any mechanism for conflicts. This proposal borrows LoRA-Gen's negative result as a design rule: it does not generate weights. It predicts only a few scalar strengths over fixed, trained adapters.
The closest routing work is V-RoLoRA. It steers a mixture of LoRA experts toward target human values, but the condition is a binary vector, and the authors name the lack of free-form or graded language as a limitation. LoraHub mixes adapters with weighted sums, but it fits the weights from few-shot examples by gradient-free search, not from instructions. MoLoRA and LoRA-Mixer route per token from hidden states, with no user instruction involved.
The motivating evidence comes from controllable generation. CocoSciSum fully fine-tunes FLAN-T5 on single-attribute data mixed together (length-only and keyword-only examples). The resulting model obeys joint length-and-keyword instructions it never saw during training. Its authors also note that it has no way to resolve contradictory constraints, such as a tiny token budget combined with many required keywords. That open problem is the target here.
Two further papers supply the design. ParSEL, working on 3D shape editing, argues that language is good at expressing the intent of an edit while a numeric parameter should carry its magnitude. We adopt that split. The orthogonality constraint comes from Orthogonal Mixture-of-Expert Low-Rank Adapter for Continual Learning. That work keeps each new adapter's projection in directions unused by earlier adapters to prevent interference, but it tested this only on image classification. We use the idea for the constraints within a single instruction, not for tasks learned in sequence.
The warning from Multi-LoRA Composition for Image Generation also shapes the design. In image generation, naively summing several adapters' weights causes concepts to bleed into each other as the number of adapters grows, and scheduling adapters across denoising steps beat merging with a win rate near 70% on 5-LoRA sets. We therefore treat interference as the main risk and measure it directly, not assume it away.
Core Mechanism
For each controllable attribute $j$ there is one adapter $B_j A_j$, trained only on single-attribute data. The attributes in this study are target length, required keywords, and forbidden keywords. At a frozen layer with weights $W_0$ and input $x$, the adapted output is
$$h = W_0 x + \sum_{j} \alpha_j(u)\, B_j A_j x,$$
where $u$ is the user's instruction and $\alpha_j(u) \ge 0$ is the strength given to attribute $j$. During adapter training we constrain the rows of each $A_j$ to be orthogonal to those of the other adapters, so the adapters write into separate input directions.
A small controller computes the strengths. A frozen sentence encoder embeds $u$. Attention over the instruction's tokens gives one summary per attribute, and a small network maps each summary to $\alpha_j$ together with a priority weight. The controller is trained in two stages. First, on single-attribute instructions paired with numeric targets, the loss asks the generated output to hit the target: length error for the length adapter, keyword inclusion for the keyword adapter. This teaches the controller how phrases like "about sixty words" or "much shorter" map to strength. Second, on a small set of paired instructions that state priorities in words, it learns how phrasing such as "above all" or "if possible" scales strengths when constraints clash. The adapters stay frozen during both stages.
This mechanism rests on a testable hypothesis: output length and keyword coverage change monotonically, and roughly separably, with $\alpha_j$. We check this first.
Research Program
The work proceeds in three steps. The first measures dose-response. For each trained adapter, we sweep $\alpha_j$ with the others held fixed and record how the target attribute and the other attributes move, both with and without the orthogonality constraint. If effects are neither monotone nor separable even under orthogonality, the design fails early and cheaply, and we report that as the result.
The second step trains the controller and evaluates three kinds of control. Graded control uses instructions that vary only in the amount requested. Conflict control uses instructions that combine incompatible constraints with an explicit priority. Unseen compositions add a third attribute, keyword exclusion, and test all three attributes jointly after training only on single attributes.
The third step checks transfer to a second domain. We convert V-RoLoRA's binary value conditions into free-form, graded value instructions and test whether per-value adapters with language-predicted strengths keep V-RoLoRA's accuracy while accepting instructions like "argue mainly from security, with some weight on tradition."
Baselines
The naive reference point is the instruction-tuned base model given the full multi-constraint instruction with no adaptation. CocoSciSum reports this FLAN-T5 baseline at ROUGE-1 33.59 and a keyword success rate of 0.24, and says it largely ignores length.
The strongest published system on this data is CocoSciSum itself: full fine-tuning on mixed single-attribute data. It reports ROUGE-1 35.35, length Pearson correlation of 0.83–0.85, keyword success rate 0.58, and human-judged factual correctness of 0.65 against 0.44 for the baseline. We rerun it on the same backbone and splits. Our method counts as beating it only if it matches it within noise on these joint-constraint numbers and clearly exceeds it on conflict adherence and graded control.
The most dangerous competitor is simpler still. It is one LoRA adapter, with the same total parameter count as our adapters combined, trained on exactly the same mixed single-attribute data with the instruction in the prompt. It is CocoSciSum's recipe done efficiently. It gets the same data, the same priority-phrased examples, the same number of training steps, and the same hyperparameter search budget. If it equals our method on conflicts and graded control, the modular claim is false.
We also run Text-to-LoRA retrained on our instruction set. This tests whether generating one whole adapter from the instruction suffices. We also run LoRA Merge of our own attribute adapters with equal, hand-set weights, which tests whether learned strengths matter at all.
Three ablations isolate the key ingredients. The first removes orthogonality. The second replaces per-clause strengths with a single vector predicted from the whole instruction. The third fixes every strength at one, which reduces the method to plain adapter addition.
Why not just…?
Why not just fine-tune one adapter with the instruction in the prompt? This is the real threat. CocoSciSum's full fine-tuning result shows that models trained this way already compose constraints they have not seen together. Our bet is narrower: this approach has no explicit handle for magnitude or priority, so under conflict it should blend constraints unpredictably and respond unevenly to graded wording. The single-adapter baseline above settles the question directly. If it matches our method on the conflict and graded suites, we drop the modular design and report that the prompt is sufficient.
Why not just generate the whole adapter from the instruction, as Text-to-LoRA does? Text-to-LoRA's authors report sensitivity to phrasing and task-level granularity. LoRA-Gen reports that direct weight prediction overfits. Predicting a few scalars over fixed adapters is a much smaller output space, which should be more robust to paraphrase. We test this by measuring how much outputs vary across paraphrases of the same instruction.
Why not just add the attribute adapters with fixed weights? Weight merging is known to interfere as the number of adapters grows, per the image-composition study. It also cannot express "a little" or "above all." The LoRA Merge baseline and the fixed-strength ablation measure how much of any gain comes from learned, instruction-dependent strengths.
Proposed Experiments
Data comes from the CocoSciSum suite. We use LEN-data from MuP (18,934 instances) for length and KW-data from SciSumm (915 instances) for keywords. We derive forbidden-keyword data from SciSumm, evaluate keyword success on SciERC KW-test (229 pairs), and run human factuality ratings on FACT-test (48 instances). From these sources we build two new evaluation suites. The conflict suite pairs very short targets with many required keywords, and states the priority in varied wording. The graded suite contains families of instructions that differ only in the amount requested. Backbones are FLAN-T5, for direct comparison with CocoSciSum, and one open decoder model.
Standard metrics are length correlation, keyword success rate, exclusion success rate, ROUGE-1, and human factuality. Three further metrics target the claims. Priority adherence is the fraction of conflict cases where the stated priority wins. Graded monotonicity is the Spearman correlation between requested amount and achieved attribute. Paraphrase variance is the spread of outputs across rewordings of one instruction.
The second domain uses Touché23-ValueEval and the Moral Integrity Corpus. The number to match there is V-RoLoRA's reported micro-F1 on Qwen3-8B: 0.5483 and 0.4446. Nothing in the retrieved literature reports results for free-form value instructions, so that part is exploratory.
Three checks decide the outcome. First, the dose-response sweep must show monotone, largely separable effects under orthogonality, and more cross-talk without it. Second, on the conflict and graded suites, our method must beat the matched single-adapter baseline by a margin larger than seed-to-seed variation, while staying within noise of CocoSciSum on ROUGE-1 and factuality. Third, on three-attribute compositions never seen in training, removing orthogonality or per-clause strengths should account for most of the gap. If the second check fails, the conclusion is that language inside the prompt already works as a dial, and that is worth publishing too.
References
- Text-to-LoRA: Instant Transformer Adaption
- SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass
- LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
- SEMANTIC-GUIDED LORA PARAMETERS GENERATION
- V-RoLoRA: RLVR-Driven MoE Routing for Steerable Pluralistic Alignment
- LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition
- MoLoRA: Composable Specialization via Per-Token Adapter Routing
- LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing
- CocoSciSum: A Scientific Summarization Toolkit with Compositional Controllability
- ParSEL: Parameterized Shape Editing with Language
- Orthogonal Mixture-of-Expert Low-Rank Adapter for Continual Learning
- Multi-LoRA Composition for Image Generation