Oriented Non-Radial Basis Functions for Image Coding and Analysis

Avijit Saha, Jim Christian, Dun-Sung Tang, Chuan-lin WuNeurIPS 1990

TL;DR

Standard Radial Basis Function (RBF) networks are limited by their "circular" nature, requiring a massive number of units to approximate complex, directional data. This paper introduces Oriented Non-Radial Basis Functions (ONRBF), which use rotatable, stretchable elliptical kernels to capture features more efficiently, significantly reducing the "curse of dimensionality" in image coding and function approximation.

The Problem with Standard RBFs

In the early 90s, "connectionist" models (neural networks) were primarily used as adaptive function approximators. A popular choice was the Radial Basis Function (RBF) network, which uses "local" kernels—usually Gaussians—to respond to specific regions of the input space.

However, standard RBFs suffer from two major flaws:

  1. The Curse of Dimensionality: Because RBF kernels are typically hyperspheres (circular), you need an exponential number of them to "tile" a high-dimensional space accurately.
  2. Lack of Orientation: If you are trying to model a diagonal stroke in an image, a circular kernel is a poor fit. You end up needing many small circles to approximate one long, thin line.

The authors sought a way to make these kernels more flexible so that a single hidden unit could represent a complex, oriented feature (like a brush stroke) rather than just a generic "blob."

The ONRBF Approach

The authors propose a generalization of the RBF called the Oriented Non-Radial Basis Function (ONRBF).

The Mathematical Shift

In a standard RBF, the response of a unit depends on the Euclidean distance between the input $x$ and a center $c$. The authors replace this simple distance with a more general polynomial. Specifically, they use a matrix $M$ to transform the input: $$R_\alpha = e^{-\| M_\alpha [x_1, \dots, x_d, 1] \|^2}$$

By using a $d \times (d+1)$ matrix, the network can simultaneously learn:

  • Translation: Where the feature is located (the center).
  • Scaling: How wide or narrow the feature is (the "sigmas").
  • Rotation: The orientation of the feature in space.

Feature Discovery in Images

When trained on an image, the hidden units don't just sit randomly; they "organize themselves" to align with the physical strokes of the character. A single unit becomes "responsible" for an entire oriented line segment. By allowing kernels to be elliptical and oriented, the network can cover the same "territory" with far fewer units than a standard RBF.

ONRBF modeling a Chinese character

Key Results and Performance

The paper evaluates the ONRBF on two distinct types of data: the Mackey-Glass Equation (a chaotic time-series benchmark) and Chinese Character Bitmaps.

Performance comparison between RBF and ONRBF
  • Superior Approximation: As shown in the performance table, an ONRBF network with only 10 units achieved a lower training error (0.229) than a standard RBF network using 40 units (0.236).
  • Better Generalization: The ONRBF consistently outperformed RBFs on unseen test data, showing that the elliptical kernels capture the underlying structure of the data rather than just memorizing points.
  • Parameter Trade-off: For 2D images, an ONRBF unit requires 6 parameters (to define the ellipse and orientation), compared to 4 or 5 for standard RBF variants. However, because you need significantly fewer units overall, the total parameter count for the whole network is much lower.

Limitations and Future Work

  • Computational Complexity: The number of parameters per unit grows at $O(d^2)$ relative to the input dimension $d$. While efficient for 2D images, this could become computationally expensive for very high-dimensional datasets.
  • Initialization Sensitivity: The authors note that if units are initialized randomly, the "features" discovered might be inconsistent across different training runs. They suggest that a "node deletion or region growing algorithm" is necessary to ensure the model consistently finds the most "meaningful" features for recognition tasks.
  • Higher-Order Statistics: For complex grayscale images, the authors suggest that the linear weights might need to be replaced by low-order polynomials to capture more nuanced texture information.

Summary updated

BibTeX
@inproceedings{saha1990oriented,
  author = {Avijit Saha and Jim Christian and Dun-Sung Tang and Chuan-lin Wu},
  title = {{Oriented Non-Radial Basis Functions for Image Coding and Analysis}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1990},
}
Figure