Structural Information Learning Machinery: Learning from Observing, Associating, Optimizing, Decoding, and Abstracting

Angsheng LiarXiv 2020 · arXiv

Abstract

This paper introduces Structural Information Learning Machinery (SiLeM), a mathematical framework that redefines learning as the optimization of structural information. By integrating graph theory, information theory, and computation, SiLeM provides a parameter-free approach to discovering the intrinsic laws of complex systems through the minimization of structural entropy. The framework shifts the paradigm of artificial intelligence from function approximation toward a first-principles theory of information processing.

Introduction

In the current landscape of artificial intelligence, learning is predominantly viewed as a problem of "function approximation." While deep learning has achieved remarkable empirical success, it often operates as a "black box," requiring massive labeled datasets and manual parameter tuning. This paper, authored by Angsheng Li, argues that a fundamental theoretical gap exists between Shannon’s information theory, Turing’s theory of computation, and modern machine learning.

To bridge this gap, the author proposes Structural Information Learning Machinery (SiLeM). SiLeM moves away from treating data as isolated random variables and instead treats learning as the process of eliminating "structural uncertainty" within a system. By merging computation with a new metric—structural entropy—SiLeM provides a mathematical definition of what it means to truly "learn" a law of nature.

The Theoretical Gap

The author identifies three primary limitations in our current understanding of intelligence:

  1. Shannon’s Information Theory: While excellent for measuring uncertainty in random variables, it fails to account for the "intrinsic information" embedded in the relationships between objects (graphs or structures).
  2. Turing Machines: These define the limits of computation but lack a formal mathematical theory for real-world information processing.
  3. Modern Deep Learning: These models lack a clear mathematical definition of learning, often relying on heuristic choices for learning rates, layer counts, and architectures.

Methodology: Structural Entropy and Encoding Trees

The core of SiLeM is Structural Entropy, a metric developed to measure the information hidden within a graph. Unlike standard Shannon entropy, structural entropy quantifies the uncertainty of a random walk on a structured data space.

The Encoding Tree

SiLeM uses Encoding Trees as a lossless method to represent graphs. An encoding tree acts as a hierarchical "decoder" that organizes data points into modules. Learning is achieved by finding the specific encoding tree that minimizes structural entropy. This optimal tree represents the "essential structure" or the "syntax" of the data.

The SiLeM Pipeline

The machinery operates through a five-stage process:

  1. Observing: Collecting raw data points from the environment.
  2. Associating: Constructing a "data space" (a graph) by linking points to maximize decoding information.
  3. Optimizing/Decoding: Executing algorithms to find the optimal encoding tree (the decoder).
  4. Interpreting: Transforming the decoder into a Knowledge Tree by assigning semantics to the discovered modules.
  5. Abstracting: Extracting common features to build Trees of Abstractions, enabling the machine to perform intuitive reasoning when encountering novel data.

Key Insights

Learning as Information Gain

The paper posits that the essence of learning is the elimination of uncertainty. If a machine has not reduced the structural entropy of its data space, it has not truly learned. In this framework, the optimal decoder is not just a data structure; it is the mathematical representation of the laws governing the observed system.

Parameter-Free Learning

Because SiLeM is based on optimizing a fundamental physical metric (entropy), it eliminates the need for the "hand-made" parameters that plague traditional machine learning. The structure of the model is determined by the data itself rather than the designer.

The Principle of Symmetry

A significant theoretical contribution is Theorem 6.3 (The Principle of Symmetry), which proves that "Compressing Information" is equivalent to "Decoding Information." This provides a foundation for big data analysis: any information lost during compression can be losslessly recovered if the correct encoding tree is identified.

Empirical Validation

While primarily theoretical, SiLeM has been validated through several high-impact applications in genomics and medicine:

  • Lymphoma Gene Expression: The machinery successfully identified tumor types and subtypes with high accuracy.
  • Hi-C DNA Folding: The "deDoC" method, built on SiLeM principles, was the first to identify Topologically Associating Domains (TADs) in single-cell biology, succeeding where traditional statistical methods failed due to data sparsity.
  • Clinical Consistency: In various cancer datasets, SiLeM identified subtypes that showed high consistency with medical labels without manual intervention.

Discussion and Future Work

SiLeM represents a significant step toward a unified theory of intelligence, yet the author acknowledges several limitations. The machine is currently "observation-dependent," meaning it cannot reason about objects it has not seen. Furthermore, there remains a gap between learning (extracting laws) and intelligence (creating new information).

Future research directions include the development of a "structural game theory" to understand how agents create new information and the application of this framework to complex human-centric tasks such as computer vision and natural language understanding. Ultimately, SiLeM suggests that by navigating trees of abstractions, machines can move beyond logical computation toward a form of global, intuitive reasoning.

Summary updated

Abstract

In the present paper, we propose the model of {\it structural information learning machines} (SiLeM for short), leading to a mathematical definition of learning by merging the theories of computation and information. Our model shows that the essence of learning is {\it to gain information}, that to gain information is {\it to eliminate uncertainty} embedded in a data space, and that to eliminate uncertainty of a data space can be reduced to an optimization problem, that is, an {\it information optimization problem}, which can be realized by a general {\it encoding tree method}. The principle and criterion of the structural information learning machines are maximization of {\it decoding information} from the data points observed together with the relationships among the data points, and semantical {\it interpretation} of syntactical {\it essential structure}, respectively. A SiLeM machine learns the laws or rules of nature. It observes the data points of real world, builds the {\it connections} among the observed data and constructs a {\it data space}, for which the principle is to choose the way of connections of data points so that the {\it decoding information} of the data space is maximized, finds the {\it encoding tree} of the data space that minimizes the dynamical uncertainty of the data space, in which the encoding tree is hence referred to as a {\it decoder}, due to the fact that it has already eliminated the maximum amount of uncertainty embedded in the data space, interprets the {\it semantics} of the decoder, an encoding tree, to form a {\it knowledge tree}, extracts the {\it remarkable common features} for both semantical and syntactical features of the modules decoded by a decoder to construct {\it trees of abstractions}, providing the foundations for {\it intuitive reasoning} in the learning when new data are observed.
BibTeX
@misc{li2020structural,
  author = {Angsheng Li},
  title = {{Structural Information Learning Machinery: Learning from Observing, Associating, Optimizing, Decoding, and Abstracting}},
  year = {2020},
  eprint = {2001.09637},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.09637},
}
Figure