Companies and governments usually measure innovation by what goes in (research spending) or by a crude count of what comes out (granted patents). The two often fail to line up. Spending rises while productivity stalls, and grant counts correlate with profits for some firms but not others. One reason is that a granted patent says little about how much of the original invention survived examination. Many applications are narrowed until they are commercially hollow, and many others are abandoned. We propose to measure this lost yield directly from the filing text. We continue training a language model on the full population of U.S. patent applications, rejected and abandoned ones included, with each document tagged by its eventual outcome. The model then gives every new filing a filing-time quality score: how much more the text resembles applications that were granted than applications that failed.
The payoff runs in two directions. For language modeling, it tests a question no existing patent model has asked. Patent models are trained almost entirely on granted text. Does adding failed applications, marked as failures, teach the model distinctions it cannot learn from successes alone? For innovation economics, it gives a text-based output measure that exists years before citations or grants accumulate. We can then ask whether a firm's average filing quality explains why its R&D spending does or does not turn into financial returns, beyond what patent counts explain.
Core Claim
We claim that outcome-tagged continual pre-training on pre-grant applications gives a better filing-time predictor of prosecution outcome than two alternatives: fine-tuning a general model as a classifier on the same labels, and domain pre-training on granted text only. We further claim that the firm-level average of this score adds explanatory power for the link between R&D and later firm performance, in regressions that already control for patent counts and firm characteristics.
The score for an application $x$ is a per-token log-likelihood ratio:
$$s(x) = \frac{1}{|x|}\Big[\log p_\theta(x \mid \texttt{[GRANTED]}) - \log p_\theta(x \mid \texttt{[ABANDONED]})\Big],$$
Here $p_\theta$ is the continually pre-trained model, $|x|$ is the number of tokens scored, and the bracketed tokens are outcome tags that were prepended during training.
Both claims can fail independently, and the design keeps them separable. If a plain fine-tuned classifier matches the pre-trained scorer, the language-modeling claim fails but the economic measurement can still be built on the classifier. If both predict prosecution outcomes well but neither adds anything in firm regressions, the economic claim fails.
Research Program
Motivating evidence. The measurement problem on the economics side is well documented. Understanding Innovation to Drive Sustainable Development predicts national innovation scores from macro indicators and argues that spending metrics capture effort rather than realized progress. The Impact of Innovation on the Financial Performance of Pharmaceutical Enterprises in the Digital Era finds that granted patents predict return on assets overall (coefficient 0.297). The effect disappears for state-owned firms (−0.028) and small firms (0.051), and patent counts alone cannot say why. The open problems in the file name the missing stage explicitly: econometric studies look at grants years after filing and never see the examination phase in which claims are narrowed or abandoned.
Source of the missing stage. The Harvard USPTO Patent Dataset (HUPD) supplies it. It contains 4.5 million applications filed from 2004 to 2018, with full pre-grant text and grant, rejection, and abandonment outcomes. Its own acceptance-prediction baselines are shallow fine-tuned encoders, and it leaves the long descriptions (around 11,800 tokens) unmodeled. Patent-CR shows what happens during examination: claims that survive become denser and less readable, and general-purpose editing models destroy that precision. We use this finding as a warning, discussed below, rather than as training data.
Training recipe. The method comes from two continual pre-training papers. D-CPT Law fits how loss depends on model size, token count, and the share of domain text in the training mix. It reports $R^2 > 0.97$ across six domains and a harmful regime below roughly a 5% domain share. We fit that law on small Qwen-1.5 runs (0.5B–1.5B) to choose the patent mixture ratio and the token budget for a 7B run. This also tests whether patent prose, which is highly repetitive and very long, follows the same curve. Efficient Continual Pre-training for Building Domain Specific Large Language Models shows that the top 10% of a financial corpus, chosen by surrogate-model perplexity and part-of-speech diversity, beats the full corpus. We apply its task-agnostic selector within each outcome class, so that selection does not quietly remove most of the failures.
Long documents. Claims and abstract are scored in full. Descriptions are split into chunks, and the score is averaged over those chunks, with the averaging weights learned on a development split. We report claims-only and description-only scores separately, since the two may carry different signal.
What would be different. On Predictive Patent Valuation forecasts citations with point processes over word-count and topic features. Patent Citation Dynamics Modeling via Multi-Attention Recurrent Networks forecasts citations from citation, assignee, and inventor timelines without reading text. Both work after grant and need early citations. BERT-CNN and Can Large Language Models Generate High-quality Patent Claims? treat patents as classification or drafting tasks and never tie model outputs to firm outcomes.
The claim-drafting paper also constrains us. There, general Llama-3 and GPT-4 beat the domain-pretrained PatentGPT-J-6B, so we cannot assume that domain pre-training helps. We therefore start from a modern base model and compare it against the same base with no further pre-training. Missing vs. Unused Knowledge Hypothesis shows that general models already hold patent-relevant knowledge that prompting can surface. That is exactly why prompting is one of our baselines.
Economic linkage. We match HUPD assignees to publicly listed U.S. firms. The firm financial data comes from outside this file, and the name matching is a real engineering cost. For each firm-year we build the count of filings, the grant rate, and the mean and upper-tail quality score $\bar s_{f,t}$. We then estimate firm and year fixed-effects regressions of performance $k$ years later (return on assets, and a productivity measure) on R&D spending, patent counts, and $\bar s_{f,t}$. The test is whether $\bar s$ changes the R&D-to-performance slope, split by firm size in the spirit of the pharmaceutical study's heterogeneity results.
Baselines
Naive reference. The simplest reference predicts the majority outcome within each technology class and filing year. At the firm level, the matching reference is the standard econometric specification with R&D spending, grant counts, size, and fixed effects and no text at all. This is the model any text measure must improve on.
Strongest existing text method. This is a fine-tuned encoder classifier on HUPD acceptance prediction, the benchmark's own baseline. The Harvard USPTO Patent Dataset reports 60–64% accuracy for BERT, DistilBERT, and RoBERTa. We rerun it on our time-based split and report both accuracy and AUC.
Strongest competitor we construct. The stronger competitor is the same 7B base model fine-tuned as a classifier head on the same labeled applications. It gets the same total compute as our continual pre-training plus scoring pipeline and the same hyperparameter search budget. Our method counts as beating it only if it improves AUC with a bootstrap confidence interval that excludes zero on held-out 2013–2015 filings.
Classical text baseline. We also include word-count novelty features (TF-IDF distance to earlier filings in the same class) fed to gradient-boosted trees, following the feature style of the citation-valuation paper.
Ablations of the key ingredient. Three ablations isolate what matters. The first pre-trains on granted applications only and scores by likelihood under that model. The second pre-trains on all applications without outcome tags and trains a probe on its embeddings. The third keeps the tags but drops perplexity-diversity selection. If the granted-only ablation matches the full model, then including failures, the central language-modeling hypothesis, is refuted.
Why not just…?
Why not just fine-tune a general model as a classifier? This is the most serious threat. The claim-drafting results suggest a strong general base plus light tuning often wins. We do not argue around it; the matched-compute classifier is the primary comparison. Our reason to expect a gain is that about 4.5 million applications give continual pre-training far more text than a classifier head can absorb. We also expect the likelihood ratio to read the full description, while a classifier usually truncates it. If the classifier ties, we report that pre-training is unnecessary and carry the classifier's score into the economic analysis.
Why not just use patent counts and citations in fixed-effects regressions? The pharmaceutical panel already shows that counts carry no signal for small and state-owned firms. Citations arrive years late, which is the cold-start problem the citation-valuation work was built to address. Still, a count-plus-citation model could explain firm performance as well as our score once a few years have passed. The regression with both sets of variables settles this: if $\bar s$ loses significance once three-year forward citations are included, its value is limited to early warning. That would be a narrower but still honest result.
Why not just prompt a large general model, with self-questioning, to rate filings? The missing-versus-unused-knowledge study shows that self-generated clarifying questions help general models tell similar patents apart. It also shows that 7B models give inconsistent answers when a question is flipped. We run this prompting baseline on a 5,000-application stratified sample with GPT-4-class and 7B open models, and measure both AUC and consistency when framing is inverted. Rating millions of filings this way costs far more than scoring them, so it would have to win clearly on accuracy to be preferred.
Proposed Experiments and Decisive Checks
We train on HUPD filings from 2004–2012 and test on 2013–2015 filings. We exclude later years because many of those applications were still pending at the dataset's 2018 cutoff, which would bias the outcome labels.
Check 1: prosecution prediction. Does the outcome-tagged model beat the matched-compute classifier and the granted-only ablation on held-out AUC? We report results within technology sections, so that the score cannot win simply by learning which fields have high grant rates.
Check 2: the readability confound. Patent-CR found that surviving claims are denser. We regress $s(x)$ on standard readability and claim-length features and test whether the residual still predicts outcome. If the score mostly rewards legalese, it is measuring drafting style rather than invention quality. That distinction matters for any economic reading of the score.
Check 3: scaling-law fit. Do D-CPT fits from the small runs predict the 7B patent loss within the reported precision? Does the harmful low-share regime appear near 5% for patent text?
Check 4: the economic test. In fixed-effects regressions, does $\bar s_{f,t}$ predict return on assets and productivity at horizons $k = 2$ to $4$ beyond counts, grant rates, and R&D? Does it carry the signal for small firms, where counts carry none? No reported result was found on Lacuna for a text-derived filing-quality measure in U.S. firm-level regressions, so the effect size here is genuinely open.
A positive answer to Checks 1 and 4 together would show that part of the input-output gap is visible in the filing text, years before grant counts or citations can reveal it.
References
- Understanding Innovation to Drive Sustainable Development
- The Impact of Innovation on the Financial Performance of Pharmaceutical Enterprises in the Digital Era
- The Harvard USPTO Patent Dataset: A Large-Scale, Well-Structured, and Multi-Purpose Corpus of Patent Applications
- Patent-CR: A Dataset for Patent Claim Revision
- D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models
- Efficient Continual Pre-training for Building Domain Specific Large Language Models
- On Predictive Patent Valuation: Forecasting Patent Citations and Their Types
- Patent Citation Dynamics Modeling via Multi-Attention Recurrent Networks
- BERT-CNN: a Hierarchical Patent Classifier Based on a Pre-Trained Language Model
- Can Large Language Models Generate High-quality Patent Claims?
- Missing vs. Unused Knowledge Hypothesis for Language Model Bottlenecks in Patent Understanding