High Order Eigentensors as Symbolic Rules in Competitive Learning

Hod Lipson, Hava T. SiegelmannHybrid Neural Systems 1998 · DOI

TL;DR

This paper introduces "high-order geometric neurons" that go beyond simple point representations to model complex shapes and topological structures in data. By using high-order tensors, these neurons can learn symbolic rules and effectively decompose data clusters, even those that are very close or overlapping, outperforming traditional methods.

What Problem Does This Paper Address?

Traditional neural networks, particularly competitive learning models like Self-Organizing Maps (SOMs), excel at mapping data to a lower-dimensional space and revealing cluster proximity. However, they often struggle to explicitly extract the underlying "laws" or symbolic structure governing the data. While SOMs can reveal topological properties, this information is often implicitly encoded in the network's weights and difficult to extract directly. The paper aims to develop neuronal units that can not only map data but also explicitly represent and extract local topological and geometrical properties, thereby uncovering symbolic rules within the data. This is particularly challenging for complex data arrangements, such as closely packed or partially overlapping clusters, which are hard for standard "point" neurons to differentiate.

Methods

The core innovation is the introduction of high-order geometric neurons. Instead of using simple weight vectors, these neurons employ high-order tensors to represent their synaptic weights. This allows them to model complex geometric shapes (like ellipsoids, and even more complex curves) rather than just points.

Key methodological aspects include:

  • Homogeneous Coordinates: Data and weights are represented in homogeneous coordinates, which elegantly incorporates both linear (first-order) and constant (zero-order) elements, simplifying the mathematical formulation.
  • High-Order Tensors: The synaptic weights are represented by tensors of increasing order (m-th order). For instance, a second-order neuron uses a covariance matrix (R), while higher-order neurons use tensors (Z) that capture higher-order correlations.
  • Eigentensors: The paper proposes analyzing the "eigentensors" of these synaptic tensors. These eigentensors are analogous to eigenvectors of a covariance matrix and reveal the coefficients of polynomial rules that the neuron has learned.
  • Maximum-Correlation Activation: Neurons are activated based on a maximum-correlation criterion, which, when adapted for higher-order tensors, effectively measures the "distance" of an input from the learned geometric shape.
  • Hebbian-Type Learning: The learning rule is a simple, local Hebbian-type update for the synaptic tensor, making it computationally efficient.

Datasets

The paper evaluates the proposed method on the 4-dimensional IRIS dataset. It notes that this dataset consists of 150 flowers across three species. The authors also present results on synthetic data clusters to demonstrate the method's ability to handle complex topological arrangements.

Key Insights and Ideas

  • Geometric Shapes as Symbolic Rules: The fundamental idea is that geometric shapes themselves can represent symbolic rules. The parameters of these shapes (captured by tensor components) directly encode these rules.
  • Decomposition of Topology: High-order neurons can decompose complex data structures, including the identification of overlaps and ambiguities, which is crucial for understanding symbolic relationships.
  • Eigentensors as Key Examples: The eigentensors of the synaptic tensor are identified as a minimal set of "key examples" that can be used to induce a learned rule, offering a direct path to symbolic rule extraction.
  • Generalization of Classic Neurons: The proposed framework elegantly generalizes classic neurons. A first-order neuron corresponds to a point, a second-order neuron to an ellipsoid (Mahalanobis metric), and higher orders capture increasingly complex shapes and curvatures.
  • Handling Overlapping Clusters: A significant advantage is the ability to correctly decompose and classify data points even when clusters are very close or partially overlap, a task that often "confuses" classic neural networks.

Key Results

  • IRIS Dataset Performance:
    • Unsupervised: The 2nd-order unsupervised network misclassified 4 out of 150 IRIS data points, and the 3rd-order unsupervised network misclassified 3.
  • Supervised: The 4th-order supervised network achieved 0 misclassifications, and higher-order supervised networks (5th to 7th order) also achieved 0 misclassifications with just one training epoch. This performance is competitive with or superior to other methods listed.
  • Qualitative Results: Figures demonstrate the network's ability to self-organize and decompose point sets into natural groups, even with significant overlap (Figure 3). It also shows the interpretation of various cluster topologies like strings, holes, and forks (Figure 4).

Limitations and Future Work

  • Network Size Selection: The paper acknowledges that determining the optimal number of neurons for a given task remains an open question.
  • Computational Cost: The increased degrees of freedom in high-order tensors can lead to higher computational costs, which needs further investigation.
  • Network Initialization: The initialization strategy for these high-order neurons is not detailed.
  • Relationship between Tensor and Shape: Further research is needed to fully understand the relationship between the values of the synaptic tensor and the specific shapes it can acquire.
  • Practical Implementation: While the paper describes the implementation and demonstrates results, further practical considerations like network initialization and cost implications are areas for future work.

Summary updated

BibTeX
@inproceedings{lipson1998high,
  author = {Hod Lipson and Hava T. Siegelmann},
  title = {{High Order Eigentensors as Symbolic Rules in Competitive Learning}},
  booktitle = {Hybrid Neural Systems},
  year = {1998},
  doi = {10.1007/10719871\_20},
}
Figure