Reasoning-Intensive Regression

Diane Tchuindjo, Omar Khattab

Introduction

Large Language Models (LLMs) have revolutionized natural language processing, yet they face significant hurdles in Reasoning-Intensive Regression (RiR). Unlike standard regression tasks that rely on shallow textual features, RiR requires deep deduction and precise numerical output—tasks such as math error detection or rubric-based essay grading. This paper introduces MENTAT, a framework designed to bridge the gap between reasoning and regression.

The MENTAT Framework

MENTAT (Mistake-Aware prompt Evolver with Neural Training And Testing) addresses the limitations of both small-model fine-tuning and zero-shot LLM prompting through a two-phase approach:

  1. Iterative Prompt Evolution: MENTAT utilizes a "batch-reflective" process where the LLM evaluates its own performance on training data, identifies error patterns, and iteratively refines its instructions. This process is roughly 8x faster than traditional evolutionary searches.
  2. Multi-Rollout Neural Aggregation: To mitigate "quantization bias"—the tendency of LLMs to output round numbers (e.g., 0.5 or 0.7)—MENTAT generates multiple independent predictions (rollouts). A lightweight Multi-Layer Perceptron (MLP) then aggregates these rollouts, learning to weight statistical features like mean and standard deviation into a calibrated final score.

Benchmarking RiR

The authors evaluated MENTAT across four diverse tasks: Mathematical Error Detection, Instruction Following, Pairwise RAG Comparison, and Essay Grading.

Distribution of ground truth scores for the Math Error Regression task, showing a spread that challenges models to avoid mean-prediction shortcuts.

The Mathematical Error Detection task is particularly challenging, as it requires the model to pinpoint the exact step where a proof fails. As shown in the distribution above, the scores are spread across the 0-10 range, preventing models from relying on simple frequency biases.

Results and Analysis

The experimental results highlight a stark contrast between traditional fine-tuning and the MENTAT approach. Small encoders like NeoBERT often fail to learn the underlying logic, instead "hacking" the loss function by predicting the mean score for all inputs—a phenomenon that results in a near-zero Concordance Correlation Coefficient (CCC).

Performance comparison on Math Error Detection. MENTAT significantly outperforms both fine-tuned small models and detailed manual prompting.

As illustrated in the figure above, MENTAT achieves a CCC of 0.792, a massive improvement over the 0.008 achieved by fine-tuned NeoBERT. Even compared to a high-quality manual prompt (CCC 0.685), MENTAT's neural aggregation and evolved prompts provide a significant edge, reducing variance by up to 30% on math tasks.

Conclusion

MENTAT demonstrates that combining LLM reasoning with neural calibration can overcome the inherent biases of language models in regression tasks. While inference costs are higher due to multiple rollouts, the 65% performance gain and increased stability make it a powerful tool for complex scoring tasks. Future work will focus on adaptive rollout strategies to optimize the balance between cost and precision.

Summary updated

Abstract

AI researchers and practitioners increasingly apply large language models (LLMs) to what we call reasoning-intensive regression (RiR), i.e., deducing subtle numerical scores from text. Unlike standard language regression tasks, e.g., for sentiment or similarity, RiR often appears instead in ad-hoc problems such as rubric-based scoring, modeling dense rewards in complex environments, or domain-specific retrieval, where much deeper analysis of context is required while only limited task-specific training data and computation are available. We cast four realistic problems as RiR tasks to establish an initial benchmark, and use that to test our hypothesis that prompting frozen LLMs and finetuning Transformer encoders via gradient descent will both often struggle in RiR. We then propose MENTAT, a simple and lightweight method that combines batch-reflective prompt optimization with neural ensemble learning. MENTAT achieves up to 65% improvement over both baselines, though substantial room remains for future advances in RiR.
BibTeX
@inproceedings{tchuindjo2025reasoning,
  author = {Diane Tchuindjo and Omar Khattab},
  title = {{Reasoning-Intensive Regression}},
  booktitle = {ER @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=m3DbJMbpXg},
}
Figure