Safety versus performance: How multi-objective learning reduces barriers to market entry

Meena Jagadeesan, Michael I. Jordan, Jacob Steinhardt

TL;DR

This paper argues that the need for AI models to balance performance with safety can actually lower barriers to market entry for new companies. By developing a multi-objective learning framework, they show that incumbents, facing greater reputational risk from safety failures, are forced to compromise on performance, allowing new entrants to compete with less data.

What Problem Does This Paper Address?

The rapid rise of large language models (LLMs) and other powerful AI systems has led to concerns about market concentration. The prevailing theory is that incumbent companies, with their vast resources for data and compute, can train superior models, creating insurmountable barriers for new entrants. This paper challenges this intuition by focusing on a crucial, often overlooked factor: safety. Incumbent AI providers face significant reputational damage if their models exhibit safety violations (e.g., generating harmful content, exhibiting bias). This pressure to ensure safety, even at the cost of peak performance, creates a unique dynamic that can be exploited by new companies.

Methods

The authors introduce a novel multi-objective high-dimensional linear regression framework to model this market entry problem. This framework captures the trade-off between a model's performance (primary objective) and its safety (secondary objective). Companies are modeled as optimizing for performance while adhering to a safety constraint, with reputational damage occurring if this constraint is violated.

Key methodological contributions include:

  • Formalizing Reputational Damage: The model explicitly incorporates the risk of reputational damage as a constraint on the incumbent's optimization process.
  • Multi-Objective Scaling Laws: They derive new scaling laws for high-dimensional linear regression in multi-objective settings. These laws describe how model performance (loss) and excess loss (performance relative to an ideal infinite-data model) improve as the dataset size increases. A key finding here is that the scaling rate slows down in multi-objective environments compared to single-objective ones, especially with larger datasets.
  • Market Entry Threshold Characterization: Using these scaling laws, they mathematically define and analyze the "market-entry threshold" – the minimum data a new company needs to outperform the incumbent.

Datasets

Not discussed in the paper. The analysis is theoretical and relies on simulated data distributions and covariance structures.

Key Insights and Ideas

  • Safety as a Market Equalizer: The most significant insight is that the "alignment tax" (the performance cost of ensuring safety) can be a powerful force for reducing market concentration. Incumbents are penalized more heavily for safety failures, forcing them to dedicate resources to safety that could otherwise go to performance.
  • Slower Scaling for Multi-Objective Learning: The derived scaling laws reveal that achieving improvements in multi-objective settings is inherently slower than in single-objective ones, especially as datasets grow. This means that the advantage of sheer data scale for incumbents is diminished when safety is a critical concern.
  • New Entrants Can Leverage Weaker Constraints: New companies, facing less scrutiny and thus weaker safety constraints, can afford to focus more on performance, allowing them to enter the market with substantially less data than the incumbent might suggest based on performance alone.
  • Regulators' Double-Edged Sword: While increased scrutiny on dominant firms can foster competition, it might also lead to a "race to the bottom" for safety if smaller firms exploit their lower reputational risk.

Key Results

The paper presents several key theoretical results:

  • Theorem 1 (Infinite Data Incumbent, Unconstrained Entrant): A new company can enter the market with finite data, even if the incumbent has infinite data, provided safety and performance are not perfectly correlated and the incumbent faces a non-trivial safety constraint.
  • Theorem 4 (Finite Data Incumbent): When the incumbent has finite data, the market-entry threshold scales sublinearly with the incumbent's dataset size. This means a new company needs proportionally less data as the incumbent's dataset grows, demonstrating a significant reduction in entry barriers. The scaling exponent can take on up to three different values, indicating complex scaling behavior.
  • Theorem 5 (New Company with Safety Constraint): Even when the new company also faces a safety constraint, it can still enter the market if its constraint is strictly weaker than the incumbent's. The market-entry threshold scales more rapidly as the safety constraints become closer, implying that tighter safety alignment between companies increases the data requirements for new entrants.

Limitations and Future Work

The authors acknowledge several limitations:

  • Simplified Marketplace Model: The model assumes homogeneous users interested in the same data distribution and that companies only compete on model performance, not price.
  • Linear Regression Assumption: The analysis is based on linear regression, which, while effective for capturing scaling trends, might not fully represent the complexity of real-world LLMs.
  • Focus on Entry, Not Survival: The paper characterizes market entry but not the long-term survival or competitive dynamics of companies.
  • Trade-off Between Competition and Safety: The authors highlight the tension between fostering market entry and ensuring overall safety, suggesting that regulators need to carefully balance these objectives.

Future work directions include:

  • Exploring Heterogeneous Users: Investigating how user heterogeneity and specialization could further impact market entry.
  • Broader Safety Objective Models: Extending the framework to more complex safety objectives beyond linear regression.
  • Empirical Validation: Empirically verifying the multi-objective scaling laws and market entry predictions for real-world LLMs.
  • Investigating Downstream Providers: Analyzing how market entry is affected when services are built on top of foundation models.
  • Alternative Optimization Objectives: Examining scenarios where companies optimize a weighted sum of performance and safety, rather than a constrained objective.

Summary updated

Abstract

Emerging marketplaces for large language models and other large-scale machine learning models appear to exhibit market concentration, which has raised concerns about whether there are insurmountable barriers to entry in such markets. In this work, we study this issue from both an economic and an algorithmic point of view, focusing on a phenomenon that reduces barriers to entry. Specifically, an incumbent company risks reputational damage unless its model is sufficiently aligned with safety objectives, whereas a new company can more easily avoid reputational damage. To study this issue formally, we define a multi-objective high-dimensional regression framework that captures reputational damage, and we characterize the number of data points that a new company needs to enter the market. Our results demonstrate how multi-objective considerations can fundamentally reduce barriers to entry-the required number of data points can be significantly smaller than the incumbent company's dataset size. En route to proving these results, we develop scaling laws for high-dimensional linear regression in multi-objective environments, showing that the scaling rate becomes slower when the dataset size is large, which could be of independent interest.
BibTeX
@inproceedings{jagadeesan2025safety,
  author = {Meena Jagadeesan and Michael I. Jordan and Jacob Steinhardt},
  title = {{Safety versus performance: How multi-objective learning reduces barriers to market entry}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2025},
  doi = {10.1073/pnas.2510004122},
}
Figure