19 Big Data and the Computational Social Science of Entrepreneurship and Innovation

Ningzi Li, Shiyang Lai, James EvansOpenalex Unknown 2025 · DOI · arXiv · OpenAlex

Abstract

As large-scale social data explode and machine-learning methods evolve, scholars of entrepreneurship and innovation face new research opportunities but also unique challenges. This paper discusses the difficulties of leveraging large-scale data to identify technological and commercial novelty, document new venture origins, and forecast competition between new technologies and commercial forms. It suggests how scholars can take advantage of new text, network, image, audio, and video data in two distinct ways: through precision measurements that function as system-level observatories and through "digital doubles" that form laboratories for virtual experimentation.

Introduction

The study of entrepreneurship and innovation has traditionally been constrained by the limitations of small-scale, longitudinal data. However, we are entering an era where "Big Data" is no longer just a collection of spreadsheets but a high-dimensional representation of human society. This paper argues for a paradigm shift: moving from using big data as a simple measurement tool ("data as data") to building "digital doubles"—generative models that act as both high-resolution observatories and virtual laboratories. By coupling massive unstructured datasets with deep learning, researchers can finally quantify elusive concepts like technological novelty and simulate the "creative destruction" of markets before they happen.

The Three Pillars of Entrepreneurial Complexity

Traditional social science struggles with three "special challenges" inherent to innovation that traditional methods are ill-equipped to solve:

  1. The Novelty Burden: Identifying true novelty requires a near-complete characterization of the existing landscape. One cannot define what is "new" without a comprehensive map of the "old."
  2. Data-Poor Origins: Startups are most critical at their inception, yet this is when they are most opaque. They lack the financial reporting of public firms and pivot too rapidly for traditional longitudinal studies.
  3. Functional Equivalence: Predicting which technology will displace another—such as an AI tool replacing a specific SaaS platform—is historically difficult until the displacement has already occurred.

While Big Data offers a solution, the common practice of "distilling" this data into simple variables often erases the nuance required to address these problems. The authors argue for a transition from traditional statistical analysis to a model-based approach.

Methodological Shift: From Statistics to Digital Doubles

The proposed framework utilizes two primary pillars to transform how we study the entrepreneurial ecosystem:

High-Dimensional Embeddings

Rather than counting keywords, researchers can project text (patents, pitches), images (founder headshots), and networks into a shared geometric space. In this embedding space, the "distance" between a new startup and existing industries provides a precise, automated measure of novelty and functional similarity. This allows for the maintenance of the "strength of weak ties" in network analysis without relying on arbitrary thresholds to clean up the data.

Digital Doubles and Generative Simulations

Using Large Language Models (LLMs) and Deep Neural Networks (DNNs), researchers can create "digital doubles" of the entrepreneurial ecosystem. These models do more than describe the past; they can be "prompted" to simulate counterfactuals. This enables virtual experimentation on how specific policy changes or funding rounds might ripple through a market. Because LLMs are pre-trained on vast societal concepts (Wikipedia, the web), they can infer deep relationships for a startup even when provided with minimal initial data, such as a short description on Crunchbase.

Key Insights: Novelty and Socio-Cognitive Paths

A transformative idea presented is viewing Novelty as "Model Surprise." By using LLM "perplexity," researchers can measure innovation: if a pre-trained model finds a venture description highly improbable or "surprising," it serves as a quantifiable signal of true novelty.

Furthermore, the research highlights that discovery is not just about the "text" of an idea, but the "path" it takes through human hands. Simulating "socio-cognitive random walks"—how researchers move between ideas and collaborators—yields significantly higher predictive power than analyzing ideas in isolation.

Empirical Results and Applications

The efficacy of these methods is demonstrated through several key findings:

  • Predicting Innovation: In a study of material discovery (e.g., COVID therapeutics), modeling the socio-cognitive process via random walks on a hypergraph improved discovery predictions by 400% compared to models using text alone.
  • Uncovering Hidden Bias: A GAN-based study on bail decisions revealed that models could identify visual dimensions like "grooming" and "heavy-facedness." These dimensions predicted outcomes as strongly as gender, capturing nuance that traditional dimensionality reduction would have erased.
  • Functional Equivalence: By projecting patent claims into embedding spaces, researchers can calculate the "cosine distance" between technologies to predict which ones are functional substitutes, enabling a predictive science of creative destruction.

Limitations and Future Directions

Despite the power of these models, several challenges remain. The rise of predictive models may trigger an Ethical Arms Race, where actors use AI to generate or filter misinformation to "win" pitches. Additionally, Interpretability remains a hurdle; the "hidden variables" within DNNs are often difficult for human theorists to translate into simple economic laws.

The next frontier lies in Multimodal Integration—moving beyond text to models that simultaneously process the "vibe" of a founder's video pitch, the technical specifications of their patent, and the sentiment of employee reviews. By coupling big data with big models, the field can move toward a more robust, predictive science of entrepreneurship.

Summary updated

Abstract

As large-scale social data explode and machine-learning methods evolve, scholars of entrepreneurship and innovation face new research opportunities but also unique challenges. This chapter discusses the difficulties of leveraging large-scale data to identify technological and commercial novelty, document new venture origins, and forecast competition between new technologies and commercial forms. It suggests how scholars can take advantage of new text, network, image, audio, and video data in two distinct ways that advance innovation and entrepreneurship research. First, machine-learning models, combined with large-scale data, enable the construction of precision measurements that function as system-level observatories of innovation and entrepreneurship across human societies. Second, new artificial intelligence models fueled by big data generate 'digital doubles' of technology and business, forming laboratories for virtual experimentation about innovation and entrepreneurship processes and policies. The chapter argues for the advancement of theory development and testing in entrepreneurship and innovation by coupling big data with big models.
BibTeX
@inproceedings{li2025big,
  author = {Ningzi Li and Shiyang Lai and James Evans},
  title = {{19 Big Data and the Computational Social Science of Entrepreneurship and Innovation}},
  booktitle = {Openalex Unknown},
  year = {2025},
  doi = {10.1515/9783111085722-019},
}
Figure