Operationalising the Definition of General Purpose AI Systems: Assessing Four Approaches

Risto Uuk, Carlos Ignacio Gutierrez, Alex TamkinarXiv 2023 · DOI · arXiv

Introduction

The European Union's landmark AI Act faces a critical implementation challenge: defining "General Purpose AI Systems" (GPAIS). While the concept of a versatile AI system is intuitive, translating it into enforceable legal language is fraught with difficulty. Current draft definitions are often criticized for being either "over-inclusive"—potentially regulating basic statistical tools like linear regression—or too vague for practical compliance. This ambiguity has created a "compliance fog"; research indicates that 13% of EU startups are uncertain if their products qualify as GPAIS, while nearly half claim the designation despite lacking clear inclusion criteria.

The Definition Dilemma

The core of the regulatory hurdle lies in the lack of concrete, operational criteria. Existing definitions from various bodies and researchers vary significantly in their scope and focus, as shown in the table below.

Table 1: Comparison of GPAIS definitions from regulatory and academic bodies.

To resolve this, the authors propose a framework centered on operationalising "distinct tasks." By moving away from binary definitions toward a multi-dimensional assessment of versatility, regulators can more accurately distinguish high-risk, general-purpose systems from narrow applications.

A Multi-Lens Framework for Classification

The paper evaluates four conceptual lenses that stakeholders can use to identify GPAIS. Rather than proposing a new algorithm, it provides a policy-oriented methodology for classification:

  1. Quantity: Identifying systems based on the raw number of distinct tasks they can perform. For instance, a system might be classified as GPAIS if it performs more than 10 tasks from an official, standardized list.
  2. Performance: Moving beyond theoretical capability to measure how effectively a system completes tasks using standardized benchmarks like HELM (Holistic Evaluation of Language Models).
  3. Adaptability: Assessing how easily a system can be "re-purposed" for new tasks via few-shot or zero-shot learning, without needing substantial architectural changes. This is a key differentiator for modern foundation models.
  4. Emergence: Identifying systems that develop "qualitative shifts" in behavior—new, unforeseen abilities—simply by scaling up parameters, data, or compute.

From Theory to Practice: Task Repositories

Operationalising these lenses requires reliable datasets to serve as "official lists" of tasks. The authors identify several existing resources that could be leveraged by regulators to create a standardized "task-counting" mechanism.

Table 2: Existing frameworks and benchmarks suitable for operationalising AI task classification.

These resources, such as O*NET for occupational tasks and BIG-bench for language model capabilities, provide the granular data necessary to move from abstract "generality" to measurable "versatility."

Strategic Risks and Policy Implications

The transition to operational definitions is not without risk. The authors highlight several critical areas for concern:

  • The "Gaming" Risk: If regulators set a hard numerical threshold (e.g., "10 tasks makes you a GPAIS"), developers might intentionally limit their systems to 9 tasks to avoid stricter scrutiny, potentially misrepresenting actual capabilities.
  • The "Linear Regression" Trap: Without specific criteria, the EU's current wording is broad enough to accidentally regulate basic statistical methods, placing an "onerous" burden on creators of simple software.
  • Emergence as a Red Flag: Emergence is a salient marker for GPAIS because it implies "lurking" capabilities that even developers might not realize exist until deployment.

Conclusion

While each of the four approaches has merit, they are most effective when used holistically. Quantity is straightforward but requires constant updates; performance benchmarks ensure meaningful capability; adaptability is the strongest differentiator for modern AI; and emergence serves as a vital indicator of unforeseen power. Implementing these approaches will require the EU to establish specialized benchmarking authorities to maintain and update task lists in the face of rapid technological advancement.

Summary updated

BibTeX
@misc{uuk2023operationalising,
  author = {Risto Uuk and Carlos Ignacio Gutierrez and Alex Tamkin},
  title = {{Operationalising the Definition of General Purpose AI Systems: Assessing Four Approaches}},
  year = {2023},
  eprint = {2306.02889},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.02889},
}
Figure