Do Reasoning Chains Help Researchers Choose, or Only Persuade Them?

Tools now recommend baselines and datasets for a new research idea and attach a step-by-step written justification, a "chain of thought", for each suggestion. These tools are judged almost entirely offline. The usual tests are whether the system would have retrieved what past authors actually used, and whether raters or a language model grading the output like the justifications. Neither test asks the question that matters to a researcher buried in thousands of papers: when I read this justification, am I better at telling a good recommendation from a bad one? A justification can make a researcher accept every suggestion more readily without helping them reject the wrong ones. If so, offline ratings would climb while real decisions got no better, which is the offline-to-online gap recommender systems already know from e-commerce.

We propose to measure this gap directly for scientific resource recommendation and then close it with a better offline metric. The metric is grounded-claim precision: split each justification into atomic factual claims (for example "paper X evaluated baseline B on dataset D"), then check each claim against a citation-and-results graph built from the literature. We hypothesize three things. First, this metric predicts how well live researchers discriminate between good and planted-bad recommendations. Second, the metrics in current use predict only how often researchers adopt recommendations: human opinion scores, chain-of-thought LLM judges, and Recall@k. Third, once the metric is validated against live behavior, it gives a cheap and trustworthy way to settle a question the literature leaves open, which is whether domain-specific pre-training yields more faithful justifications than prompting a general model.

Core Claim

The unit of online success for a justified recommendation is not the click or the rating. It is calibrated adoption: the probability that a researcher adopts a good recommendation minus the probability that they adopt a bad one. Offline metrics that reward fluency and plausibility will track overall adoption. Only a metric that checks justification content against verifiable evidence will track calibrated adoption.

The motivating evidence comes from three sources. AgentExpt recommends baselines and datasets using a citation-context retriever and a reasoning reranker. The reranker is a fine-tuned DeepSeek-R1-Distill-Qwen-7B that writes natural-language justifications. AgentExpt reports gains in Recall@20 and HitRate@10 plus a mean opinion score from 15 researchers, but it never measures what researchers do with the output. SciArena shows that the best automated judges agree with live researcher preferences only about 65.1% of the time on literature-grounded tasks. ALERT documents a "salesy trap" in which language-model explanations score well on engagement-style offline criteria while being promotional; its chain-of-thought judge reached 70% concordance with experts but was never tested online.

The proposal is also constrained by other results. Gaming the Judge shows that unfaithful reasoning chains can fool model-based evaluators, so no LLM judge can serve as ground truth here. Do Offline Metrics Predict Online Performance in Recommender Systems? found high rank correlation (above 0.8) between offline and online metrics in its RecLab simulations, but also a plateau beyond which offline gains stopped mattering. We therefore expect conventional metrics to correlate with online outcomes coarsely, and to fail at the fine distinction between persuading a researcher and informing them.

SPECTER pre-trained a document embedder on citation graphs and is one of the few scientific systems validated online, reporting a 46.5% click-through improvement in a production A/B test. Click-through, however, is exactly the adoption-only signal we argue is insufficient once justifications enter the picture.

Research Program

The first component is an evidence graph. We reuse the AgentExpt construction: citation contexts from 108,825 papers across 10 AI conferences, linking papers to 116,970 baselines and 68,316 datasets. We add results tables extracted in the style of LLMEVALDB, which records which method was reported on which dataset with which metric. A claim extractor splits each justification into atomic claims. Each claim is labeled supported, contradicted, or unverifiable against the graph. The claim extractor and the verifier are audited on a hand-labeled sample before use. Grounded-claim precision is the fraction of claims that are supported. We report it together with the unverifiable rate, so that a system cannot score well by saying nothing checkable.

The second component is a fixed ranking with varied justifications. To isolate the effect of the justification, every condition shows the same ranked list, produced by an AgentExpt-style retriever and reranker. Only the accompanying text changes. There are four live conditions. The first shows no justification. The second shows extractive evidence, meaning the verbatim citation-context snippets and table rows that link the resource to similar work. The third shows the reasoning chain written by the reranker. The fourth is a deliberately persuasive control, written by the same model prompted to argue enthusiastically for each item without adding facts. The persuasive control reproduces ALERT's salesy failure on purpose, to test whether each offline metric can detect it.

The third component is planted-error recommendations. In each participant's list, a fixed share of items are degraded substitutes: a dataset from the wrong modality or task, a baseline superseded on the relevant benchmark, or a pairing the graph shows was never compatible. An expert panel certifies these as clearly worse, blind to condition. Justifications are generated for planted items in the same way as for real ones. The planted items are what let us measure discrimination rather than mere adoption.

The fourth component is domain pre-training as the downstream question. Once the metric is validated, we compare three justifiers using offline measures only. The first is the general 7B reasoning model with chain-of-thought prompting. The second is the same model after continued pre-training on citation contexts and full-text method sections from the evidence corpus, then given the same reranker fine-tuning. The third is a general frontier model prompted with retrieved context. This directly tests the tension the file identifies. On one side, the PEFT comparison for code and MonoCoder argue that domain pre-training matters more than adaptation method. On the other, systems like SurveyMaster get strong LLM-judged scores from prompted general models. Our contribution is to answer this with a metric shown to track what researchers do, rather than with a judge.

Baselines

The do-nothing reference is the ranked list with no justification. It sets the floor for calibrated adoption. If researchers already reject planted items reliably from titles alone, justifications have little room to help. It runs on the same ranking, participants, and items as every other condition.

The strongest existing method is the AgentExpt reasoning reranker with its own justifications, which is our third condition. AgentExpt reports Recall@20 up 5.85% and HitRate@10 up 7.90% over SymTax and DataFinder, with a mean opinion score of 4.12/5 for baseline recommendations and 4.38/5 for dataset recommendations (AgentExpt). We retrain it on the same graph, with the same temporal cutoff and compute budget as all our variants. We do not claim to beat its ranking, since the ranking is shared across conditions. The claim instead is that our metric predicts its live calibrated adoption better than its opinion score does.

The strongest existing offline evaluator is an ALERT-style chain-of-thought pairwise judge, scoring Reasoning, Clarity, Engagement, and Neutrality. ALERT reports 70% concordance with experts (ALERT), and SciArena reports about 65.1% judge accuracy against researcher preferences (SciArena). We run it with a current frontier model, tuning its prompt on a held-out split with the same number of attempts we allow our claim extractor. Grounded-claim precision beats it if its correlation with per-item calibrated adoption is significantly higher across conditions and items, and if it ranks the persuasive control below the reasoning-chain condition where the judge does not.

The ablation that isolates the key ingredient is the extractive-evidence condition. It supplies the same verifiable facts with no generated reasoning. If it matches the reasoning-chain condition on calibrated adoption, the benefit comes from exposing evidence, not from the chain itself.

Why not just…?

Why not just trust an LLM judge with chain-of-thought, as SurveyMaster and ALERT do? The file's evidence says such judges disagree with researchers about a third of the time (SciArena), and Gaming the Judge shows they can be swayed by plausible but unfaithful reasoning. The persuasive control is a direct test. If the judge scores it at or above the factual reasoning chains while researchers' discrimination drops under it, the judge is measuring persuasiveness. If the judge instead tracks calibrated adoption as well as our metric does, we will report that and the metric adds nothing.

Why not just show a good ranked list and skip justifications, as SPECTER-style recommenders do? This is a real possibility: ALERT's finding that similarity-based justifications have near-zero win rates concerns rater preference, not decision quality. The no-justification condition settles it. If calibrated adoption is no worse without text, the honest conclusion is that justifications are cosmetic for this task. That would still be worth knowing, and it would still show opinion scores to be misleading.

Why not just show the retrieved evidence instead of generating reasoning? This is the most serious threat, and we expect it may win on discrimination. Our core claim survives if it does: extractive evidence has grounded-claim precision near one by construction, so it should rank first on our metric. The metric would then correctly predict that generated chains add persuasion without information. What would falsify the proposal is extractive evidence scoring high on our metric but low on live discrimination.

Why not just fine-tune a general model with lightweight adapters instead of domain pre-training? The code-generation comparison (PEFT study) suggests pre-training dominates, but that is a different domain and outcome. We include an adapter-only variant of the 7B model, matched in tuning budget, in the offline justifier comparison. Its grounded-claim precision decides the question here.

Proposed Experiments

The live study recruits roughly 60 machine-learning researchers, mostly graduate students and postdocs. Each describes a current project in a paragraph and receives 12 baseline and dataset recommendations, of which 3 are planted. Conditions are randomized at the block level within each participant, to gain within-subject power while limiting carry-over between styles. The immediate measure is which items participants add to their experiment plan. A follow-up after three to four weeks records which items they actually ran or still intend to keep, a behavioral outcome closer to what SPECTER's click-through test captured, but one that separates good from bad adoption. The primary outcome per condition is calibrated adoption.

The study is pre-registered, and we will state its power honestly. With about 45 planted items per condition, it can detect large discrimination differences, not subtle ones. A pilot will determine whether the design needs to be cut to three conditions.

In parallel, we compute every offline metric on the same items: Recall@20 and HitRate@10 under a temporal split (training graph frozen before the participants' project dates), mean opinion scores from a separate rater pool, ALERT-style judge scores, and grounded-claim precision.

The first decisive check is the predictive comparison: at the level of individual items and conditions, which offline metric best predicts calibrated adoption, and which predicts only raw adoption. The second decisive check is the persuasive control. Our hypothesis is supported if it raises raw adoption and judge scores, lowers calibrated adoption, and is ranked last by grounded-claim precision alone. The third check, run only if the first two succeed, applies the validated metric to the general, adapter-tuned, domain-pre-trained, and frontier-prompted justifiers on the full temporal test set. We add a cold-start slice restricted to resources released after the cutoff, since AgentExpt's reliance on citation contexts makes such items its known weak point. This gives a behaviorally anchored answer to whether domain pre-training makes reasoning chains more faithful, instead of one that rests on a judge researchers disagree with a third of the time.