TL;DR
DUALVIEW is a parameter-efficient framework for multi-output text regression that outperforms state-of-the-art baselines while using up to 1,800x fewer trainable parameters. By extracting both "broad" and "focused" evidence from frozen Large Language Models (LLMs), it provides a general-purpose way to predict multiple numerical scores—like code efficiency or research novelty—simultaneously and accurately.
What Problem Does This Paper Address?
In the world of LLMs, we often want to evaluate text along multiple dimensions. For a piece of generated code, you might care about correctness, runtime, and memory usage. For a research paper, you might want to score its novelty, soundness, and presentation. This is called Multi-Output Text Regression (MOTR).
Existing approaches to MOTR face three major hurdles:
- Cost: Fine-tuning an entire LLM for every new regression task is computationally expensive and doesn't scale.
- Complexity: Training a separate predictor for every single output dimension (e.g., one for "fluency," one for "factuality") increases the number of parameters linearly with the number of targets.
- Domain Specificity: Many current methods rely on specialized architectures (like graph neural networks) that work well for essay scoring but fail when applied to code or dialogue.
The authors of DUALVIEW set out to create a lightweight, general-purpose method that shifts the focus from complex architectures to smarter "evidence extraction" from frozen, pre-trained models.
Methods
DUALVIEW uses a dual-branch architecture to extract complementary information from a frozen LLM without changing the LLM's underlying weights.
- The Broad View: This branch queries the LLM with a prompt describing all target metrics. It mean-pools the hidden states to get a "big picture" representation. To keep things efficient, it uses reduced-rank regression, which identifies common predictive patterns shared across all outputs.
- The Focused View: This branch uses specific prompts for each target (e.g., "Assess only the runtime"). It extracts the hidden state from the very last token of these prompts. Interestingly, it uses a shared linear predictor across all these specialized embeddings, pooling supervision across targets to learn a robust scoring function.
- Efficiency Trick: To avoid redundant computation, DUALVIEW uses shared-prefix reuse. Since the broad and focused prompts share the same input text and instructions, the model caches these states and only processes the unique "suffixes" for each focused query.
Datasets
The researchers evaluated DUALVIEW across five diverse tasks to prove its general-purpose nature:
- MSumBench: Summarization quality (completeness, conciseness, faithfulness).
- RATE: Machine translation (accuracy, fluency, style).
- CreativeEval: Research ideation (contribution, soundness, presentation, overall score).
- DSCodeBench: Code generation metrics (correctness, runtime, peak memory).
- PRISM: Dialogue alignment (values, safety, creativity, etc.).
Key Insights and Ideas
- Evidence Complementarity: The "aha" moment is that broad context and target-specific focus provide different, additive value. Ablations showed that using both views significantly outperforms using either one alone.
- Frozen is Better: By freezing the LLM, DUALVIEW avoids the "catastrophic forgetting" or instability often seen when fine-tuning large models for regression. It proves that the necessary "evidence" for scoring is already present in the LLM's internal representations; you just need to extract it correctly.
- Small Supervised > Giant Unsupervised: One of the most striking findings is that DUALVIEW built on a 31B open-weight model (Gemma) outperformed zero-shot and few-shot prompting with a frontier model like Gemini 3.1 Pro.
Key Results
DUALVIEW demonstrated dominant performance across the board:
- Accuracy: It improved aggregate Pearson correlation by 18.4% over the strongest supervised baseline (GAT-AES).
- Consistency: As shown in the radar plots below, DUALVIEW (the solid teal line) consistently achieves the highest correlation across almost all 21 individual output targets.
- Efficiency: It requires only 5.1K to 57.4K trainable parameters. For context, the strongest baselines required between 82x and 1,842x more parameters to achieve worse results.
- Data Efficiency: The model maintains its lead even when training data is scarce, outperforming baselines with as few as 64 examples.
Limitations and Future Work
While powerful, DUALVIEW has some constraints:
- White-box Requirement: Because it relies on extracting internal hidden states, it cannot be used with "black-box" API models (like GPT-4) where only text output is available.
- Prompt Sensitivity: Like most LLM-based methods, the quality of the extracted evidence can be sensitive to how the prompts are formatted.
- Inference Scaling: While parameter-efficient, the computational cost at inference still grows linearly with the number of outputs (though this is mitigated by prefix caching).
Future work may involve extending this framework to distribution elicitation (predicting a range or probability distribution of scores rather than just a single number) and integrating multi-objective optimization to better balance conflicting targets during training.
Summary updated