Directional-Unit Boltzmann Machines

Richard S. Zemel, Christopher K. I. Williams, Michael MozerNeurIPS 1992 · OpenReview

Introduction

The Directional-Unit Boltzmann Machine (DUBM) extends the classic binary Boltzmann machine to handle continuous, cyclic data such as angles and phases. By representing units as complex numbers on a unit circle, the model naturally captures periodic information and provides a built-in measure of "certainty" through its mean-field approximation.

The Problem with Binary Units

Standard Boltzmann machines are restricted to binary states (0 or 1). However, many real-world variables are inherently periodic or directional—such as wind direction, the day of the week, or the orientation of an edge in an image. Mapping these continuous, cyclic variables onto binary units is inefficient and destroys the underlying topology of the data.

Furthermore, traditional neural units often struggle to represent "confidence." The authors sought a framework where a unit could not only represent a value (like a specific angle) but also communicate how certain the network is about that value based on the agreement of its inputs.

The Directional-Unit Boltzmann Machine

To address these limitations, the authors introduce the DUBM, which generalizes the Boltzmann machine using complex variables.

  • State Representation: Each unit $j$ has a state $z_j = e^{i\tau_j}$, representing a point on the unit circle with a phase $\tau_j$ between $0$ and $2\pi$.
  • Complex Weights: Weights $w_{jk}$ are also complex numbers. To ensure a well-defined energy function, the weight matrix $W$ is constrained to be Hermitian ($W^T = W^*$), the complex-domain equivalent of a symmetric weight matrix.
  • Energy Function: The network utilizes a Hermitian quadratic energy function. In a stochastic DUBM, the probability of a unit's state follows the von Mises distribution (often called the "circular normal" distribution).
Mean-field representation of a directional unit
  • Mean-Field Approximation: To make the model computationally efficient, the authors derive a deterministic "mean-field" version. In this mode, a unit’s state $y_j$ is a vector inside the unit circle. Its phase represents the mean direction, while its magnitude (resultant length $r_j$) represents the degree of certainty.
  • Learning Algorithm: The paper generalizes the standard Boltzmann learning rule using a gradient-based approach. It minimizes the difference between two states: one where input and output units are "clamped" to training data, and one where only inputs are clamped.

Key Insights

  • Certainty as Magnitude: One of the most elegant features of the DUBM is that the magnitude of the net input naturally describes how much the various inputs "agree." If inputs predict wildly different phases, they cancel out, leading to a small magnitude (low certainty). If they align, the magnitude is maximized (high certainty).
  • Global Rotation Invariance: Because the energy function depends on the relative phases between units rather than absolute values, the entire network's state can be rotated without changing its energy. This is highly advantageous for tasks like image segmentation, where the absolute phase representing "Object A" is irrelevant, provided all pixels in Object A share the same phase.
  • Principled Circular Statistics: Unlike previous "rotor" models that relied on ad-hoc derivations, the DUBM grounds its mean-field equations in the formal mathematics of the von Mises distribution.

Experimental Validation

The paper evaluates the DUBM across three primary tasks, demonstrating its versatility:

  1. Associative Memory: A 30-unit fully connected network was used to test pattern completion. The DUBM successfully recovered stored patterns from corrupted inputs. Thanks to its rotation invariance, it could also recognize and complete rotated versions of those patterns.
  2. Non-Linear Mapping (Robot Arm Kinematics): The network was tasked with mapping two input angles (joint positions) to one output angle (the angle of the line connecting the base to the fingertip). A compact DUBM with only 8 hidden units learned this complex trigonometric relationship. On novel test patterns, it achieved a mean angular error of less than 0.05 radians, accompanied by a high certainty measure ($r > 0.85$).
  3. Image Segmentation (MAGIC): In a large-scale application, the model learned to assign object labels to image features (like edge segments) based on consistent phases. The "certainty" feature allowed the system to ignore initially random phases and only propagate labels once a consensus (high $r$) began to emerge, successfully segmenting overlapping geometric objects.

Limitations and Future Directions

While powerful, the DUBM shares the capacity constraints of the Hopfield networks it generalizes; its autoassociative memory performance degrades quickly if too many patterns are stored.

Looking forward, the authors suggest that real-world problems might benefit from hybrid architectures combining binary units (to represent the presence or absence of a feature) with directional units (to represent its orientation). Additionally, they propose applying DUBMs to time-series prediction, specifically for fluctuating directional data like wind patterns.

Summary updated

Abstract

We present a general formulation for a network of stochastic di(cid:173) rectional units. This formulation is an extension of the Boltzmann machine in which the units are not binary, but take on values in a cyclic range, between 0 and 271' radians. The state of each unit in a Directional-Unit Boltzmann Machine (DUBM) is described by a complex variable, where the phase component specifies a direction; the weights are also complex variables. We associate a quadratic energy function, and corresponding probability, with each DUBM configuration. The conditional distribution of a unit's stochastic state is a circular version of the Gaussian probability distribution, known as the von Mises distribution. In a mean-field approxima(cid:173) tion to a stochastic DUBM, the phase component of a unit's state represents its mean direction, and the magnitude component spec(cid:173) ifies the degree of certainty associated with this direction. This combination of a value and a certainty provides additional repre(cid:173) sentational power in a unit. We describe a learning algorithm and simulations that demonstrate a mean-field DUBM'S ability to learn interesting mappings. Many kinds of information can naturally be represented in terms of angular, or directional, variables. A circular range forms a suitable representation for explicitly directional information, such as wind direction, as well as for information where the underlying range is periodic, such as days of the week or months of the year. In computer vision, tangent fields and optic flow fields are represented as fields of oriented line segments, each of which can be described by a magnitude and direction. Directions can also be used to represent a set of symbolic labels, e.g., object label A at 0, and object label B at 71'/2 radians. We discuss below some advantages of representing symbolic labels with directional units.
BibTeX
@inproceedings{zemel1992directional,
  author = {Richard S. Zemel and Christopher K. I. Williams and Michael Mozer},
  title = {{Directional-Unit Boltzmann Machines}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1992},
  url = {https://openreview.net/forum?id=BkZRVubd-B},
}
Figure