Introduction
Large Language Models (LLMs) have revolutionized natural language processing, yet they face significant hurdles in Reasoning-Intensive Regression (RiR). Unlike standard regression tasks that rely on shallow textual features, RiR requires deep deduction and precise numerical output—tasks such as math error detection or rubric-based essay grading. This paper introduces MENTAT, a framework designed to bridge the gap between reasoning and regression.
The MENTAT Framework
MENTAT (Mistake-Aware prompt Evolver with Neural Training And Testing) addresses the limitations of both small-model fine-tuning and zero-shot LLM prompting through a two-phase approach:
- Iterative Prompt Evolution: MENTAT utilizes a "batch-reflective" process where the LLM evaluates its own performance on training data, identifies error patterns, and iteratively refines its instructions. This process is roughly 8x faster than traditional evolutionary searches.
- Multi-Rollout Neural Aggregation: To mitigate "quantization bias"—the tendency of LLMs to output round numbers (e.g., 0.5 or 0.7)—MENTAT generates multiple independent predictions (rollouts). A lightweight Multi-Layer Perceptron (MLP) then aggregates these rollouts, learning to weight statistical features like mean and standard deviation into a calibrated final score.
Benchmarking RiR
The authors evaluated MENTAT across four diverse tasks: Mathematical Error Detection, Instruction Following, Pairwise RAG Comparison, and Essay Grading.
The Mathematical Error Detection task is particularly challenging, as it requires the model to pinpoint the exact step where a proof fails. As shown in the distribution above, the scores are spread across the 0-10 range, preventing models from relying on simple frequency biases.
Results and Analysis
The experimental results highlight a stark contrast between traditional fine-tuning and the MENTAT approach. Small encoders like NeoBERT often fail to learn the underlying logic, instead "hacking" the loss function by predicting the mean score for all inputs—a phenomenon that results in a near-zero Concordance Correlation Coefficient (CCC).
As illustrated in the figure above, MENTAT achieves a CCC of 0.792, a massive improvement over the 0.008 achieved by fine-tuned NeoBERT. Even compared to a high-quality manual prompt (CCC 0.685), MENTAT's neural aggregation and evolved prompts provide a significant edge, reducing variance by up to 30% on math tasks.
Conclusion
MENTAT demonstrates that combining LLM reasoning with neural calibration can overcome the inherent biases of language models in regression tasks. While inference costs are higher due to multiple rollouts, the 65% performance gain and increased stability make it a powerful tool for complex scoring tasks. Future work will focus on adaptive rollout strategies to optimize the balance between cost and precision.
Summary updated