Scientific paper recommendation is a problem of almost no data. Most users have saved a handful of papers, and most new papers have been saved by nobody. The one reliable signal for a brand-new paper is its abstract. That makes it natural to encode abstracts with a language model pre-trained on scientific text and then train that encoder on who saved what.
The difficulty is that the gradients from user feedback are sparse and noisy. Left to themselves, they can wear away the very domain knowledge that made the encoder worth using, and a cold-start item has nothing else to fall back on. This note proposes a simple remedy and a controlled test of it. While the encoder is trained to recommend, it also keeps training on the objective it was pre-trained with: reconstructing masked spans of each item's abstract, with sentence order left intact. The claim is that the right auxiliary task for a domain-pretrained recommender is its own pre-training task. The claim also says why: this task anchors the domain meaning of words against drift caused by recommendation training, and it helps most on new items that are also in-domain.
The idea combines two lines of work that have not met. Multi-task recommendation, meaning training shared user and item representations on several objectives at once, has shown that an extra text task regularizes sparse collaborative signals. Domain-specific pre-training has shown that corpus-level exposure to technical text beats surface fine-tuning of general models. Neither literature has asked what happens when the two are combined, or whether the combination buys anything beyond simply freezing a good domain encoder.
Core Claim
Our base model follows the item representation of Ask the GRU. There, each item's vector is a free item-specific factor plus a text encoding of its abstract. For a cold-start item the free factor is zero, so the item is represented only by its text. We replace that paper's from-scratch GRU (a small recurrent text encoder) with the encoder of a domain-pretrained sequence-to-sequence model, and we train with
$$\mathcal{L} = \mathcal{L}_{\text{rec}}(\mathbf{p}_u, \mathbf{q}_j + \mathbf{d}_j(\theta)) + \lambda\, \mathcal{L}_{\text{infill}}(\theta; \text{abstract}_j).$$
Here $\mathbf{p}_u$ is the user's latent factor and $\mathbf{q}_j$ is item $j$'s free factor. $\mathbf{d}_j(\theta)$ is the mean-pooled encoding of item $j$'s abstract under encoder weights $\theta$. $\mathcal{L}_{\text{rec}}$ is a pairwise ranking loss over saved and unsaved papers. $\mathcal{L}_{\text{infill}}$ is the span-infilling reconstruction loss computed on the same item abstracts, and $\lambda$ weights it.
We predict three things. First, this model beats a frozen domain encoder on cold-start recall. Second, it beats recommendation-only fine-tuning of the same encoder, and the gap is explained by measurable drift of the encoder's representations away from their pre-trained geometry. Third, the gain is concentrated on test items whose abstracts fall inside the pre-training domain.
Relation to Prior Work
Two papers motivate the gap, and both stop short of it. Ask the GRU couples collaborative filtering with a GRU text encoder on CiteULike. It reports that an auxiliary tag-prediction task adds about 7–8% Recall@50, but it never uses a pre-trained language model. CiteBART shows that masking-based pre-training on scientific text yields a strong citation generator, but it is unpersonalized and has no user component.
Two further papers supply the method. The domain encoder comes from BioBART, which continued BART pre-training on PubMed abstracts. It found that dropping sentence permutation and keeping only span infilling improves biomedical performance, so we reuse exactly that objective, discourse order included. Because BioBART and vanilla BART share an architecture and vocabulary, they give us a clean domain-versus-general contrast.
Two papers constrain the design. CARec documents "item representation contamination": naively backpropagating collaborative signals degrades pre-trained semantic item embeddings. Its fix is alternating alignment with a frozen general-purpose encoder (Instructor-xl), and it never tries a domain encoder or a text-reconstruction anchor. Our proposal attacks the same failure in a different way. We do not freeze the encoder. We keep its pre-training loss running.
PETER is the direct challenge. Its unpretrained two-layer Transformer beat fine-tuned BERT on explanation generation (BLEU-4 3.43 vs 0.24 on Yelp). Its authors attribute this to a mismatch between user and item IDs and word tokens. We take this seriously. PETER used generic BERT on consumer reviews, so it leaves open whether domain pre-training behaves differently on technical items. We include a from-scratch arm to settle this, not assume it.
CPRec uses "domain pre-training" in the recommender sense: continued pre-training on behavioral sequences. We deliberately keep that out of scope and treat it as an orthogonal alternative.
Research Program
The work has three parts.
The model. We attach BioBART-Base's encoder, or BART-Base's for the general control, to the hybrid item representation above. The decoder is kept only to compute the infilling loss on item abstracts. Training alternates minibatches of user–item pairs with minibatches of masked abstracts drawn from the training items' text. Cold-start test abstracts are never used, which prevents transductive leakage.
The drift analysis. We measure how far the encoder moves from its pre-trained state in two ways. The first is representational similarity between pre-trained and fine-tuned pooled encodings on held-out abstracts. The second is a linear probe that predicts CiteULike user-assigned tags from frozen encodings. The hypothesis requires that recommendation-only fine-tuning shows larger drift and lower probe accuracy than the infilling-anchored model. It also requires that, across runs and values of $\lambda$, less drift predicts better cold-start recall. If drift is unrelated to recall, the "anchor" explanation is wrong, even if accuracy improves.
The domain-match analysis. We split CiteULike test items into biomedical and non-biomedical subsets with a keyword-and-venue classifier that we validate by hand on a sample. The claim predicts that BioBART's advantage, and the anchor's advantage, both appear mainly on the biomedical subset. A uniform gain would point to generic regularization rather than domain knowledge.
Baselines
The naive reference is a content-only model. We mean-pool the frozen BART-Base encodings of abstracts, set all free item factors to zero, and learn only user factors. This shows what a general encoder gives with no adaptation. Its frozen BioBART counterpart is the first real competitor, and we discuss it below.
The strongest existing method on this exact task is the multi-task GRU of Ask the GRU. We re-implement it with its tag-prediction auxiliary on CiteULike-a and CiteULike-t, using the same cold-start protocol and the paper's Recall@50. The file reports only relative gains: up to 34% over collaborative topic regression in cold start, and 7–8% from multi-tasking. No reported absolute Recall@50 was found on Lacuna for this baseline, so our re-implementation must reproduce a comparable relative gain over its single-task variant before we trust the comparison.
We also run CARec's two-phase alignment with its encoder swapped to BioBART. This tests whether alignment against a frozen encoder already solves contamination. Finally, we run a PETER-style unpretrained Transformer of matched width, trained from scratch on CiteULike abstracts with the same infilling auxiliary.
Every arm gets the same splits, the same negative sampling, the same maximum GPU hours, and an equal hyperparameter search budget (the same number of trials over learning rate, embedding size, and $\lambda$ or its analogue). Results are averaged over five seeds.
Two ablations isolate the key ingredient. The first replaces infilling with Ask the GRU's tag prediction on the BioBART encoder, which tests whether any auxiliary works or specifically the pre-training objective. The second turns sentence permutation back on in the infilling loss, which tests BioBART's discourse-order finding in a recommender.
Beating the field means a statistically reliable cold-start Recall@50 gain across seeds over the best of three arms: frozen BioBART, BioBART with tag prediction, and CARec with BioBART. The gain must hold on both CiteULike variants.
Why not just…?
Why not just freeze a domain encoder and learn collaborative factors on top? This is the most serious threat. If the frozen encoder already carries the domain semantics, then anchoring an unfrozen one may only recover what freezing preserved for free. CARec's result that frozen semantic targets help suggests this is plausible. The experiment decides it directly. If frozen BioBART matches the anchored model on cold-start recall, we will report that the right move in sparse technical domains is to not fine-tune, and that is a useful negative. The claim survives only if fine-tuning with the anchor adds recall that neither freezing nor unanchored fine-tuning provides.
Why not just train a compact model from scratch, as PETER does? PETER's evidence comes from dense consumer-review data, where the model sees a great deal of in-distribution text per item. CiteULike's cold-start items offer one abstract each, in technical vocabulary. There, A Comparative Study of PEFT Methods for Python Code Generation found that domain pre-training predicts downstream quality more strongly than the fine-tuning method used. Still, we do not assume the answer. The matched from-scratch arm with the same infilling loss will show whether pre-training matters at this data scale.
Why not just prompt a large general LLM, or pre-train on behavior as CPRec does? CPRec itself reports that zero-shot general LLMs drop sharply on recommendation. Its fix, behavioral continued pre-training, needs dense interaction sequences across many domains that CiteULike lacks. It also addresses collaborative logic, not item semantics. We therefore treat CPRec as complementary rather than competing. We include only a zero-shot LLM ranking check on a subsample as a sanity floor.
Why not just add a contrastive objective such as SimGCL's embedding noise? SimGCL regularizes the ID embeddings toward uniformity. That touches the collaborative factors, not the text encoder that cold-start items depend on. We add it as an optional term on top of both the anchored and unanchored models, to check that the anchor's gain is not absorbed by it.
Decisive Checks
The idea stands or falls on three results on CiteULike-a and CiteULike-t cold-start splits. First, anchored BioBART must beat frozen BioBART and tag-prediction BioBART on Recall@50. Second, lower encoder drift must predict higher recall across runs. Third, the domain-versus-general advantage must concentrate on biomedical items. If the first check fails, we report that freezing suffices. If the second fails, the gain is not about preserving domain semantics. If the third fails, domain pre-training is acting as generic regularization, and the note's framing should be revised accordingly.
References
- Ask the GRU: Multi-task Learning for Deep Text Recommendations
- CiteBART: Learning to Generate Citations for Local Citation Recommendation
- BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model
- Collaborative Contextualization: Bridging the Gap between Collaborative Filtering and Pre-trained Language Model
- Personalized Transformer for Explainable Recommendation
- Large Language Model Empowered Recommendation Meets All-domain Continual Pre-Training
- A Comparative Study of PEFT Methods for Python Code Generation
- Are Graph Augmentations Necessary?: Simple Graph Contrastive Learning for Recommendation