TL;DR
By applying narrative theory to over 150 years of U.S. patent data, researchers have categorized innovations into "suspense" (anticipated connections) and "surprise" (unexpected breakthroughs). While surprising innovations are often disruptive, it is the suspenseful ones—those that society has been waiting for—that achieve the highest market value and citation impact.
What Problem Does This Paper Address?
Predicting the "next big thing" is the holy grail of R&D, but technological progress is notoriously non-linear. Most predictive models treat innovation as a steady decline in uncertainty. However, this ignores the nature of that uncertainty.
The authors argue that innovation is an information-revealing process, much like a story. Sometimes we see a plot point coming (suspense), and sometimes a twist catches us off guard (surprise). By framing innovation through these two lenses, the authors aim to explain why some groundbreaking ideas integrate seamlessly into the economy while others—despite their brilliance—struggle to find a foothold.
Methods
The researchers treated the history of technology as a massive network of connections between different domains.
- Link Prediction: They used U.S. Patent Classification (USPC) codes to represent technological domains. An "innovation" is defined as a new link between two codes that have never appeared together in a single patent before.
- Belief Scores: Using a word2vec model, they treated patents like sentences and technology codes like words. This allowed them to calculate "context similarity"—how often two technologies appear in similar environments, even if they haven't met yet.
- Categorization:
- Suspense: Occurs when the model predicts a high likelihood of a connection (a high "belief score"), but the actual link doesn't happen for years. It represents "ideas in the air."
- Surprise: Occurs when a connection is made between two technologies that the model (and by extension, society's current logic) did not expect.
Datasets
The study utilizes the United States Patent and Trademark Office (USPTO) dataset, specifically:
- 8,884,966 utility patents spanning from 1940 to 2015 (with some historical analyses reaching back to 1840).
- 438 technology codes at the 2-digit main class level.
- The analysis is limited to pre-2015 data because the USPC system was replaced by the Cooperative Patent Classification (CPC) thereafter.
Key Insights and Ideas
- The "Ideas in the Air" Reservoir: The researchers found a remarkably stable reservoir of about 20,000 "unrealized suspense" pairs at any given time. These are connections that are mathematically "due" to happen but haven't been realized by an inventor yet.
- The Edison vs. Tesla Dynamic: The paper uses these two icons to illustrate the trade-off. Thomas Edison’s patents showed a balance of suspense and surprise, reflecting his focus on business-oriented, "integratable" tech. Nikola Tesla’s work was heavily skewed toward surprise—experimental ideas that laid the groundwork for the future but were often too "novel" for his own era to fully adopt.
- The 1920 Pivot: Around 1920, the dominant pattern of technology life cycles shifted. Modern "cutting-edge" tech usually follows an on-cycle pattern: it starts with a Surprise (innovation), moves to Suspense (syndication), and ends in Popularity (diffusion). Older, foundational technologies often follow an off-cycle pattern, where popularity comes first, followed by unexpected new uses much later.
Key Results
- Impact and Value: Suspenseful innovations are significantly more likely to become "hits." They have a higher probability of ranking in the top 5% of citations and command a higher median market value.
- The Adoption Penalty: Surprising innovations, while often disruptive, face an "adoption challenge." Their hit probability and market value fluctuate below the dataset average, suggesting that extreme novelty can be a barrier to societal integration.
- Predictive Performance: The model achieved an Area Under the Curve (AUC) of 0.7660, indicating a strong ability to distinguish between likely and unlikely future technological pairings.
Limitations and Future Work
The authors acknowledge several constraints:
- Classification Rigidity: The model can only detect links between existing categories. It cannot identify the birth of entirely new technological domains that fall outside current classification codes.
- Subjectivity: Patent codes are assigned by human examiners, introducing a layer of subjective judgment into the underlying data.
- Geographic Bias: The study is limited to U.S. patents, which may not fully reflect global innovation dynamics.
Future research could expand this framework to include hyperlinks (connections between more than two technologies) or use machine learning to see how AI-driven predictions might be changing the "suspense" levels of modern R&D.
Summary updated