People ask questions in the register of social media: slang, hashtags, half-sentences. The documents that answer them are written in the register of news, law and science. Keyword search such as BM25, which ranks documents by shared words, fails whenever the two sides use different words for the same thing. Neural "query rewriters" are small language models trained to rephrase a question before it is sent to search, and they are the usual fix. But they rewrite everything in the query, including the names of people, drugs and organizations. Those names are the part keyword search already handles well. They are also the part where rewriters invent facts, because a rewriter that has seen "Pfizer" thousands of times learns to append whatever it remembers about Pfizer. This proposal splits the job. Before rewriting, we replace entity names with placeholders. We rewrite only the surrounding wording, the predicate ("got jabbed", "is suing", "dropped a new"), into the formal register. Then we put the original names back by plain string substitution. The rewriter is trained on pairs harvested continuously from the live web: informal posts paired with the formal news articles they link to. In each pair, the entity names the two sides share are masked, so the model can only learn how informal wording maps to formal wording.
The claim to test is that masking changes what the rewriter learns, and not only what it outputs. A rewriter that never sees entity names cannot learn shortcuts tied to particular entities. It should therefore keep its accuracy on queries about entities that appear after its training data was collected. An unmasked rewriter should lose accuracy on exactly those queries. Because the web supplies fresh informal–formal pairs every month, we can test this directly with a time-split evaluation that static benchmarks cannot support.
Core Claim and Relation to Prior Work
Three lines of work rewrite or expand queries to bridge vocabulary gaps, all on fully lexicalized strings. Query Rewriting for Retrieval-Augmented Large Language Models trains a T5 rewriter with reinforcement learning, using the reader's answer accuracy as the reward. RaFe trains rewriters on ranking feedback from a neural reranker, about 100 times more cheaply than feedback from an LLM. EAR samples many candidate expansions and learns to pick the one that best serves BM25. Our method reuses their machinery: we will use EAR's sample-and-rerank selection as the decoding step. The difference is that none of the three distinguish entity wording from predicate wording, and none target a gap in register as opposed to a gap in topic.
Entity masking supplies the method. Covering Cracks in Content Moderation replaced seed drug names with a mask token so that a classifier learned the "contextual fingerprint" of drug talk. The resulting system transferred from Reddit to the Silk Road forum and found about 4 times more unique jargon than keyword systems. That paper is a classifier with a fixed seed list. We apply the same idea to the generation side of retrieval. Delexicalized Paraphrase Generation constrains our design. It reports that a decoder's exact copying of slot tokens degrades as the number of slots grows. We avoid that bottleneck: placeholders are forced through decoding as fixed tokens and replaced afterwards, so the decoder never copies names itself. Mention randomization (Hongyu Lin et al.) and type masking (Danqi Chen) offer competing placeholder schemes, which we compare.
The dynamic web supplies the training signal. A Continuously Growing Dataset of Sentential Paraphrases showed that tweets sharing a news URL yield over 114,000 paraphrases from a 1% stream. Those pairs naturally cross from user commentary to edited headlines. That paper notes alignment rests on shared named entities, which is exactly why unmasked models trained on such data can match names instead of meaning. We turn that weakness into the training recipe. The shared spans are, by construction, the entities to mask, and they are found by string overlap with the formal side rather than by a named-entity tagger on noisy tweets. Chirps similarly abstracted arguments to align predicates from news tweets. More than 60% of the predicate pairs it found were absent from PPDB. That supports the claim that live harvesting covers wording a static resource misses. Chirps stopped at an offline table; we feed the same signal into a retrieval pipeline.
Two papers motivate the regime. BEIR shows dense retrievers lose to BM25 on 17 of 18 zero-shot datasets. EAR shows sparse search with selected expansions beating DPR on rare-entity queries (65.5% versus 38.1% top-5 on EntityQs). Together these say entity names are best left to exact matching, while the predicate is where help is needed. TWEETQA quantifies the register gap: only about 10% vocabulary overlap with SQuAD.
Research Program
We build three things.
The first is a harvesting pipeline. It monitors news-account links in a public social stream and collects user posts that link to an article. It aligns each post to the article's headline and lead sentences. It then masks maximal spans that occur verbatim, or nearly so after case and hashtag normalization, on both sides. Each pair is stamped with its month.
The second is a rewriter, a T5-sized sequence-to-sequence model trained on masked informal-to-formal pairs. Placeholders are indexed (ENT1, ENT2) and decoding is constrained to emit each one exactly once. We then fine-tune it with RaFe-style reranker feedback on the target collection.
The third is test-time masking for user queries, which have no linked article. We mask spans that match high-specificity n-grams in the formal collection's vocabulary, meaning terms that are rare across documents, or titles in an entity index. We report how often this matcher is wrong as a measured variable rather than an assumption.
The analysis has two parts. We count what rewriters add, classifying added terms as predicate paraphrases or entity-associated facts. We also track how retrieval quality changes with the time gap between training and test data.
Baselines
The naive reference point is BM25 on the raw informal query. It costs nothing, is robust across domains per BEIR, and fails precisely on register gaps. WikiQA found 20.3% of correct answers share no content words with the question. Any rewriter must clear BM25 on every split, and especially must not lose to it on entity-heavy queries.
The strongest existing method is EAR, run from its released code (voidism/EAR). It is the strongest competitor because it already pairs sparse search with learned expansion and is the best reported rare-entity result in the file: 65.5% top-5 on EntityQs and 69.3% top-5 on Natural Questions, against DPR's 68.3% (EAR). For fairness, EAR's expansion generator and query reranker are retrained on the same harvested pairs without masking. EAR receives the same backbone size, number of samples per query and hyperparameter search budget as our model, and uses the same BM25 index. Beating it means a significant gain in recall@20 and nDCG@10 on the post-training-window split, with no loss on EntityQs. Matching EAR in-window while beating it out-of-window is still a positive result. Losing out-of-window would falsify the claim.
We also run a RaFe-style rewriter (RaFe) trained with the same reranker feedback, and a zero-shot LLM rewriter. For the LLM rewriter, the file reports HyDE at 41.8 MAP and InteR at 50.0 MAP on TREC DL'19 (Synergistic Interplay). Those numbers are on formal web queries, so they frame the comparison but do not transfer to our task.
The key ablation trains the identical rewriter on the identical pairs with masking switched off. It isolates masking from data, architecture and decoding. Secondary ablations swap indexed placeholders for typed labels and for random strings, and replace web-harvested pairs with PPDB-derived pairs at matched size. The PPDB ablation isolates the contribution of the dynamic web.
Why not just…?
Why not just prompt a large LLM to rewrite the query? This is a real threat on register: frontier models know slang. Two observations argue it falls short. The JEDIS paper reports that GPT-4o-mini had high recall but low precision on jargon, for example reading "speed" as a drug name (Covering Cracks). An LLM also has a knowledge cutoff, while our time split deliberately tests entities and slang that postdate it. We include a zero-shot and a few-shot LLM rewriter and report entity-preservation rate alongside retrieval. If the LLM matches us on the post-cutoff split, our contribution shrinks to a cheaper, smaller model, and we will say so.
Why not just append the original query to the rewrite, so entities survive anyway? Concatenation is the standard guard against drift, and it may recover much of the benefit. But it preserves the names without stopping the rewriter from adding wrong entity-associated terms that pull BM25 toward the wrong documents. We test this directly. Both the masked and unmasked rewriters are run with and without concatenation, and we measure the share of added terms that are entity facts not present in the query. If unmasked-plus-concatenation equals masked on the time split, masking is unnecessary.
Why not just fine-tune a dense retriever on the same informal–formal pairs? In-domain, this could win. BEIR's negative result concerns zero-shot transfer, not same-distribution training. We train a DPR-style dual encoder on the same pairs with the same compute. We then compare on the time split and on transfer to a different formal collection (SciFact and TREC-COVID from BEIR), where the sparse-plus-rewrite design should hold up better. If the dense model wins out-of-window too, we report that the masking advantage is specific to sparse pipelines.
Proposed Experiments
The primary benchmark is built from the harvesting pipeline. A user post serves as the query, and the collection is a month-stamped news archive with the linked article as the gold document. Training months precede test months by gaps of one to six months. Entity novelty is labelled per query, according to whether its masked spans ever occurred in training. No reported result exists on Lacuna for retrieval on this construction. The nearest number, DeepPairwiseWord's 0.749 F1, is for paraphrase identification on the same source and is not comparable.
Secondary evaluations cover three settings. EntityQs and Natural Questions confirm masking does not hurt entity-heavy formal queries. BEIR's TREC-COVID and SciFact serve as no-harm checks. A small stress test grounds informally restated MISCIPLUS claims (Grounding Fallacies) against their source papers. That paper reports open-retrieval MRR up to 0.664 for formal claims, so we can measure how much an informal restatement costs and how much rewriting recovers.
Metrics are recall@20, nDCG@10 and MRR for retrieval, entity-preservation rate and hallucinated-entity-term rate for rewrites, and wall-clock cost per query.
The decisive check is the interaction between masking and time gap. The prediction is that the unmasked rewriter's advantage over BM25 shrinks as the gap grows and on novel-entity queries, while the masked rewriter's advantage stays roughly flat. If both curves are the same, the core claim fails. A second check reruns the ablation with gold entity spans in place of the test-time matcher, which tells us whether any shortfall comes from masking itself or from noisy span detection on informal text.
References
- Query Rewriting for Retrieval-Augmented Large Language Models
- RaFe: Ranking Feedback Improves Query Rewriting for RAG
- Expand, Rerank, and Retrieve: Query Reranking for Open-Domain Question Answering
- Covering Cracks in Content Moderation: Delexicalized Distant Supervision for Illicit Drug Jargon Detection
- Delexicalized Paraphrase Generation
- A Continuously Growing Dataset of Sentential Paraphrases
- Acquiring Predicate Paraphrases from News Tweets
- BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- TWEETQA: A Social Media Focused Question Answering Dataset
- WikiQA: A Challenge Dataset for Open-Domain Question Answering
- Synergistic Interplay between Search and Large Language Models for Information Retrieval
- Grounding Fallacies Misrepresenting Scientific Publications in Evidence