DUALVIEW: Leveraging Broad and Output-Specific Evidence for LLM-based Multi-Output Text Regression

Yiheng Su, Junyi Jessy Li, Matthew LeasearXiv 2026 · arXiv

TL;DR

DUALVIEW is a parameter-efficient framework for multi-output text regression that outperforms state-of-the-art baselines while using up to 1,800x fewer trainable parameters. By extracting both "broad" and "focused" evidence from frozen Large Language Models (LLMs), it provides a general-purpose way to predict multiple numerical scores—like code efficiency or research novelty—simultaneously and accurately.

What Problem Does This Paper Address?

In the world of LLMs, we often want to evaluate text along multiple dimensions. For a piece of generated code, you might care about correctness, runtime, and memory usage. For a research paper, you might want to score its novelty, soundness, and presentation. This is called Multi-Output Text Regression (MOTR).

Existing approaches to MOTR face three major hurdles:

  1. Cost: Fine-tuning an entire LLM for every new regression task is computationally expensive and doesn't scale.
  2. Complexity: Training a separate predictor for every single output dimension (e.g., one for "fluency," one for "factuality") increases the number of parameters linearly with the number of targets.
  3. Domain Specificity: Many current methods rely on specialized architectures (like graph neural networks) that work well for essay scoring but fail when applied to code or dialogue.

The authors of DUALVIEW set out to create a lightweight, general-purpose method that shifts the focus from complex architectures to smarter "evidence extraction" from frozen, pre-trained models.

Methods

DUALVIEW uses a dual-branch architecture to extract complementary information from a frozen LLM without changing the LLM's underlying weights.

Figure 2: DUALVIEW architecture. DUALVIEW queries a frozen LLM with a broad prompt and focused prompts to extract complementary representations. The “Broad View” jointly predicts all output dimensions from one representation using a reduced-rank predictor. On the other hand, the “Focused View” predicts each output dimension individually using a shared linear predictor with target-specific representations. The two branches predict standardized scores, which are summed and rescaled to obtain the final predictions.
  • The Broad View: This branch queries the LLM with a prompt describing all target metrics. It mean-pools the hidden states to get a "big picture" representation. To keep things efficient, it uses reduced-rank regression, which identifies common predictive patterns shared across all outputs.
  • The Focused View: This branch uses specific prompts for each target (e.g., "Assess only the runtime"). It extracts the hidden state from the very last token of these prompts. Interestingly, it uses a shared linear predictor across all these specialized embeddings, pooling supervision across targets to learn a robust scoring function.
  • Efficiency Trick: To avoid redundant computation, DUALVIEW uses shared-prefix reuse. Since the broad and focused prompts share the same input text and instructions, the model caches these states and only processes the unique "suffixes" for each focused query.

Datasets

The researchers evaluated DUALVIEW across five diverse tasks to prove its general-purpose nature:

  • MSumBench: Summarization quality (completeness, conciseness, faithfulness).
  • RATE: Machine translation (accuracy, fluency, style).
  • CreativeEval: Research ideation (contribution, soundness, presentation, overall score).
  • DSCodeBench: Code generation metrics (correctness, runtime, peak memory).
  • PRISM: Dialogue alignment (values, safety, creativity, etc.).
Figure 1: Examples of multi-output regression tasks like code generation and research ideation.

Key Insights and Ideas

  • Evidence Complementarity: The "aha" moment is that broad context and target-specific focus provide different, additive value. Ablations showed that using both views significantly outperforms using either one alone.
  • Frozen is Better: By freezing the LLM, DUALVIEW avoids the "catastrophic forgetting" or instability often seen when fine-tuning large models for regression. It proves that the necessary "evidence" for scoring is already present in the LLM's internal representations; you just need to extract it correctly.
  • Small Supervised > Giant Unsupervised: One of the most striking findings is that DUALVIEW built on a 31B open-weight model (Gemma) outperformed zero-shot and few-shot prompting with a frontier model like Gemini 3.1 Pro.

Key Results

DUALVIEW demonstrated dominant performance across the board:

  • Accuracy: It improved aggregate Pearson correlation by 18.4% over the strongest supervised baseline (GAT-AES).
  • Consistency: As shown in the radar plots below, DUALVIEW (the solid teal line) consistently achieves the highest correlation across almost all 21 individual output targets.
  • Efficiency: It requires only 5.1K to 57.4K trainable parameters. For context, the strongest baselines required between 82x and 1,842x more parameters to achieve worse results.
  • Data Efficiency: The model maintains its lead even when training data is scarce, outperforming baselines with as few as 64 examples.
Figure 3: Predictive Performance across datasets and dimensions. Each spoke reports Pearson correlation for one dataset aspect, macro-averaged over 8 LLM backbones and 3 training seeds. We compare DUALVIEW against the three strongest supervised baselines. Larger radial values are better, where scales vary across datasets for visibility.

Limitations and Future Work

While powerful, DUALVIEW has some constraints:

  • White-box Requirement: Because it relies on extracting internal hidden states, it cannot be used with "black-box" API models (like GPT-4) where only text output is available.
  • Prompt Sensitivity: Like most LLM-based methods, the quality of the extracted evidence can be sensitive to how the prompts are formatted.
  • Inference Scaling: While parameter-efficient, the computational cost at inference still grows linearly with the number of outputs (though this is mitigated by prefix caching).

Future work may involve extending this framework to distribution elicitation (predicting a range or probability distribution of scores rather than just a single number) and integrating multi-objective optimization to better balance conflicting targets during training.

Summary updated

Abstract

While work on LLM-based regression has largely focused on scalar prediction, multi-output text regression remains less explored. We present DualView, a parameter-efficient approach to leverage LLMs for multi-output text regression. \method{} first extracts broad and output-specific representations from a frozen LLM, then uses a dual-branch architecture to produce the full score vector. Across five heterogeneous regression tasks and eight LLMs (up to 122B), DualView outperforms seven supervised regression baselines while requiring 82--1842 times fewer fitted parameters than the three strongest baselines. These improvements extend across individual output dimensions and persist even with limited training data. Notably, DualView using a smaller open-weight LLM can outperform unsupervised prediction with Gemini 3.1 Pro. Controlled ablations show that broad and output-specific evidence provide complementary predictive benefits, while lightweight shared predictors achieve strong performance with fewer trainable parameters.
BibTeX
@misc{su2026dualview,
  author = {Yiheng Su and Junyi Jessy Li and Matthew Lease},
  title = {{DUALVIEW: Leveraging Broad and Output-Specific Evidence for LLM-based Multi-Output Text Regression}},
  year = {2026},
  eprint = {2610.19206},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.19206},
}
Figure