Training on Community Votes Instead of Filtering by Them

When people continue training a language model on a specialized domain, they usually treat the domain corpus as flat text and filter it beforehand. Technical community sites such as Stack Overflow and Math StackExchange come with something web text lacks: a record of how experts judged each post. That record includes upvotes, accepted-answer flags, author reputation, and comment exchanges. This proposal asks a simple question. Should those signals decide which text the model trains on, or should the model see them during training as labels and learn what separates a trusted answer from a wrong one? We propose quality-conditioned continual pre-training. A decoder-only language model keeps training on all community Q&A text, but each answer is prefixed with a short metadata tag that encodes its peer-judged quality, normalized so that old, heavily viewed threads do not dominate. A final phase without tags lets the model work on plain text. At inference time, the model can be steered toward the "accepted, high-consensus" register. Its own likelihoods can also be used to rank answers that carry no metadata.

The opportunity comes from two findings that have not been connected. In Stack Overflow answer selection, a metadata-only model beats a neural text-only model at picking the accepted answer: 0.463 versus 0.363 Precision@1, where Precision@1 is the fraction of questions whose top-ranked answer is the accepted one (Preferred Answer Selection in Stack Overflow). Social signals therefore carry quality information that current text models fail to extract. Separately, BERTOverflow, a model pre-trained on raw Stack Overflow text, underperformed general-purpose RoBERTa (PTM4Tag). That is evidence that uncurated community text can hurt a domain model. Our claim is that the community's own judgments, given to the model as conditioning rather than used only as a filter, are what make this noisy domain text worth training on.

Core Claim

Continually pre-training a decoder model on community Q&A with prefixed, normalized quality metadata should beat three alternatives under an equal token budget: plain continual pre-training, vote-threshold filtering, and a text-classifier quality filter. It should win on two measures. The first is ranking accepted answers from text alone. The second is the quality of generated answers. It should also forget no more general ability than the alternatives. We predict the gain comes from the quality content of the tags and not merely from grouping documents. Tag-based conditioning is known to help through grouping alone, so this is the claim that needs testing. If tags with the true quality meaning removed work just as well, the claim is wrong.

Relation to Prior Work

Three papers motivate the design and also set its limits.

Metadata Conditioning Accelerates Language Model Pre-training (MeCo) prepends source URLs during most of general web pre-training and ends with a phase on plain text. It matches the standard baseline with 33% fewer tokens. Its "grouping hypothesis" constrains us: hashed URLs gave comparable speedups, so the benefit may come from clustering similar documents rather than from what the metadata means. MeCo uses only coarse source identity, does no domain continual pre-training, and uses no quality signals. We move conditioning to graded within-domain quality, and we test directly whether the meaning of the tags matters.

FastDoc and its predecessor FPDM replace token-level masked prediction with triplet and hierarchical-classification losses over static taxonomies such as arXiv categories and EUR-Lex labels. They report 500x to 4,500x compute reductions. Masked-prediction training dropped about 11 points on the grammaticality task CoLA (part of the GLUE benchmark), against about 1 point for FastDoc. That work is encoder-only, uses institutional labels rather than social judgments, and names robustness to noisy metadata as open. Votes are exactly that kind of noisy metadata.

Organize the Web shows that 84% of the gain from the FineWeb-Edu quality filter is recovered by matching its topic distribution. This is a warning for us. Any apparent "quality" gain could really be a hidden change in topic mix, so our comparisons must hold the tag and topic distribution fixed.

The quality-signal side supplies the method. RedCaps curated a strong image-text corpus with a crude threshold of at least two upvotes, a binary filter we use as a baseline. Extracting Chatbot Knowledge from Online Discussion Forums and Hybrid Attentive Answer Selection in CQA show that author longevity and history predict quality. That motivates a reputation field in the prefix. The Stack Overflow study found that badges add noise and that asker–answerer comment interaction was the most useful signal, so we drop badges and encode whether the asker engaged with the answer. Its open problems, temporal volatility and manipulable votes, motivate the normalization below. On the domain side, BioBART found that disrupting discourse order hurt biomedical pre-training. We therefore serialize threads in their natural order: question, answer, comments.

Research Program

The corpus is the public Stack Overflow and Math StackExchange dumps. Each answer is serialized with its question and comment thread. Raw vote counts confound quality with exposure, because old and popular threads collect more votes. We therefore encode each answer's quality as its rank among sibling answers in the same thread, plus a coarse bucket of score divided by question views. The prefix also carries the accepted flag, a bucketed author reputation computed only from activity before the post (so no information leaks from the future), and whether the asker replied in comments. Example: [rank:1/4 | score-bucket:high | accepted | rep:mid | asker-engaged].

Following MeCo, loss is computed on text tokens only. About 90% of the continual pre-training uses prefixes, and the last 10% does not. The base model is an open roughly 1B-parameter decoder. This scale is chosen because MeCo shows gains from 600M parameters upward, and it allows many matched runs.

The model is used in two ways. The first is conditional generation under the high-quality prefix. The second is metadata-free ranking. Here a candidate answer is scored by the gap between its likelihood under the high-quality prefix and its likelihood under the low-quality prefix. This contrast asks whether the model has turned social judgment into a text-only quality detector. We also run an encoder variant that adds a vote-rank objective to FastDoc-style training, to see whether the effect depends on architecture.

Baselines

The naive reference point is the base decoder with no continual pre-training, evaluated with the same scoring and prompting.

The first real competitor is plain continual pre-training on the full serialized corpus with no prefixes. This is the BERTOverflow-style setting that PTM4Tag suggests may be harmful.

The second is RedCaps-style filtering: train only on answers with a score of at least 2, and separately only on thread-top or accepted answers. Training is repeated over the filtered set so the total token count is the same.

The third is a FineWeb-Edu-style classifier filter. A text classifier is trained to predict our quality buckets, and only its top-scored data is kept at the same token count. We found no reported result on Lacuna for a FineWeb-Edu-style filter on community Q&A.

The fourth is MeCo with tag-only prefixes, meaning the site and topic tags without quality fields. This tests whether domain grouping alone explains the gains.

The key ablation keeps our prefix format and bucket frequencies but randomly permutes the quality values across answers. Grouping structure and topic mix are preserved, but the link to actual quality is broken.

Every run uses the same base model, the same number of training tokens, the same learning-rate sweep (three values per arm), and the same evaluation code. Each arm runs with three seeds.

The strongest existing reference on answer selection is the text-plus-metadata model at 0.527 Precision@1, with text-only decatt at 0.363 and metadata-only at 0.463, on Stack Overflow LARGE (Preferred Answer Selection in Stack Overflow). On SemEval-2016 Task 3, the strongest reported system is UIA-LSTM-CNN at 0.8086 mean average precision, which uses author history (Hybrid Attentive Answer Selection in CQA). Our text-only likelihood contrast beats the field in a meaningful way if it clearly exceeds 0.363 and approaches 0.463 using no metadata at test time. It must also beat the filtered and shuffled-tag arms by more than the variation across seeds.

Why not just…?

Why not just filter by votes, as RedCaps did? This is the most serious threat. Filtering is simpler, and RedCaps shows it can work well. Our argument is that low-vote answers still teach domain vocabulary and common misconceptions, and that conditioning lets the model learn the contrast between good and bad answers rather than never seeing bad ones. Filtering also cannot produce our likelihood-contrast ranker at all. Still, if the filtered arm matches us on generation quality and general ability, the honest conclusion is that filtering is enough for generation. Conditioning would then be justified only for ranking. The matched-token comparison settles this directly.

Why not just use a generic text quality classifier? Organize the Web shows that such filters largely act as topic reweighting. The Stack Overflow study shows text-only judgment of answer quality lags metadata by about 10 points of Precision@1. A classifier trained on text alone is likely to inherit that gap. We include this classifier-filter arm and control for topic mix. If it matches us after topic matching, community metadata adds nothing beyond what text reveals.

Why not just use MeCo's URL or tag conditioning? MeCo's own hashed-URL result predicts that any consistent prefix helps by grouping. The tag-only arm and the permuted-quality ablation exist to test this. If permuted tags perform as well as true ones, our quality-semantics claim fails. In that case the contribution shrinks to "MeCo-style conditioning also helps domain continual pre-training."

Why not just fine-tune a general model with lightweight adapters on accepted answers? A Comparative Study of PEFT Methods for Python Code Generation finds that the base model's domain pre-training predicts downstream performance more strongly than the choice of adapter method. We nonetheless include a LoRA arm (low-rank adapter fine-tuning) trained on accepted answers with matched compute, as an extra check.

Proposed Experiments

Answer selection. Accepted-answer selection on Stack Overflow LARGE is measured by Precision@1, with SemEval-2016 Task 3 measured by mean average precision. Both use our text-only likelihood contrast.

Math answer retrieval. We evaluate on ARQMath Task 1 (Math StackExchange answer retrieval), re-ranking a BM25 candidate list. The file reports only Task 2 formula-retrieval numbers (NDCG' 0.555, P'@10 0.488, Approach Zero and Anserini). We found no reported result on Lacuna for ARQMath Task 1, so we report our own BM25 re-ranking baseline.

Generation. For held-out questions posted after the training cutoff, we measure the perplexity of the eventually accepted answer under each model. We also run blind pairwise preference judgments of answers generated by each arm.

Forgetting. We run the OLMES suite before and after continual pre-training. The comparison point is the forgetting that plain continual pre-training is known to cause (FastDoc).

Decisive checks. Three outcomes decide the claim. First, true quality tags must beat permuted tags on answer selection. Second, conditioning must beat vote filtering at equal tokens. Third, for answers posted after the training cutoff, the likelihood-contrast ranker must agree with the eventual accepted answer. That third test removes leakage from votes the model may have memorized.

We also check whether the model learns popularity rather than correctness. Using Math StackExchange answers that were later edited for errors or flagged in comments, we test whether high-quality conditioning prefers the corrected version. Comparing within-thread normalized tags against raw vote tags measures how much the normalization matters.