TL;DR
This paper argues that traditional engineering—which relies on breaking systems into independent parts—fails when applied to Artificial General Intelligence (AGI) because AI and its environment are inextricably linked. The authors propose a new framework based on "Core" and "Periphery" precepts, using Ashby’s Law of Requisite Variety to model intelligence as a relationship between a system and its environment rather than a standalone property.
What Problem Does This Paper Address?
Traditional systems engineering is built on the "V-model": you decompose a complex problem into smaller, independent functional components, build them, and then recompose them. This works because we can usually draw a hard line between a component's inputs and outputs.
However, AI-enabled systems (and eventually AGI) break this model. An AI influences its environment, and that environment, in turn, influences the AI. This "environmental coupling" means the boundaries between the system and its surroundings dissolve. If you can't clearly partition inputs from outputs, traditional decomposition fails. The authors highlight a "gap" in current AI engineering: we can specify the functional requirements of a Deep Neural Network (DNN) easily, but we struggle to guarantee the outcomes because those outcomes depend on a shifting, complex relationship with the world.
Methods
The authors turn to Abstract Systems Theory and Cybernetics to find a solution. Specifically, they leverage Ashby’s Law of Requisite Variety, which states that for a system to successfully regulate another (like an AI regulating its environment to achieve a goal), the "variety" (complexity/entropy) of the regulator's actions must match or exceed the variety of the disturbances it faces.
They introduce two new formal precepts:
- Core ($C$): The parts of a system’s structure that remain invariant over time. This corresponds to the stable, "mechanical" aspects of the system where traditional engineering still applies.
- Periphery ($P$): The "residual change"—the parts of the system that shift, adapt, or grow in response to the environment.
By mathematically defining these based on set differences of system states over time, the authors provide a way to track how a system adapts without needing to assume that its components are independent.
Datasets
Not discussed in the paper. This is a theoretical and conceptual paper focused on engineering philosophy and systems theory; it does not perform empirical experiments on specific datasets.
Key Insights and Ideas
- Intelligence as a Relationship: The authors posit that intelligence shouldn't be viewed as a property of the system itself, but as a property of the relation between the system and its environment.
- The Failure of the "V": Traditional "engineering by aggregation" (if the parts work and the interfaces work, the whole works) is in direct conflict with the highly coupled nature of general intelligence.
- Core vs. Periphery in DNNs: In a Deep Neural Network, the "Core" consists of the universal functions (like backpropagation or layer-to-layer information passing). The "Periphery" consists of the specific weights and training data, which are environment-dependent and do not follow "universal" engineering rules.
- Symmetry and Blocking: The framework allows engineers to analyze "symmetry"—whether a stable system is regulating a stable environment—and "blocking"—how a system uses its internal variety to cancel out environmental disturbances to achieve a desired outcome.
Key Results
The paper provides a formal mathematical foundation for the Core and Periphery:
- Definition of Residual Change ($R$): A measure of how much the system's input/output structure changes between two points in time.
- Core/Periphery Equations: $C = \bar{S} \setminus R$ and $P = R$.
- Abstraction Independence: These precepts can be applied at any level—system, subsystem, or component—allowing engineers to model adaptation without requiring full observability of the entire complex system.
Qualitatively, the paper concludes that while traditional engineering is a "precept for open systems," Core and Periphery are "precepts for closed systems" (where the system and environment are treated as a single unit), which is a more appropriate lens for AGI.
Limitations and Future Work
The authors acknowledge several hurdles:
- Empirical Validation: The framework needs to be tested on actual intelligent systems to see if it can effectively isolate functions in practice.
- Formal Rigor: While the concepts are defined, a longer-form mathematical elaboration involving theorems and corollaries is still needed.
- The Boundary Paradox: Even though the paper argues that boundaries dissolve, engineers still need to define some temporary boundaries to perform measurements. Research into the "dynamics of boundaries" in highly coupled systems is a suggested next step.
Summary updated