Multiplicative Updating Rule for Blind Separation Derived from the Method of Scoring

Howard Hua Yang

Introduction

In Blind Source Separation (BSS), the primary objective is to recover independent source signals from a set of observed mixtures without prior knowledge of the mixing process. Standard gradient descent operating in Euclidean space is often inefficient for this task because the parameter space of demixing matrices possesses a non-flat geometry.

To address this, researchers previously developed the relative gradient and natural gradient to make updates "equivariant"—ensuring that the algorithm's performance remains consistent regardless of how poorly scaled the underlying mixing matrices are. However, there remained a need for a more rigorous statistical foundation for these methods. This paper addresses whether the Method of Scoring (a classic statistical estimation technique) can be used to derive these efficient, equivariant rules and whether the approach can be simplified for practical, high-dimensional applications.

Methodology

The author leverages Information Geometry to treat the space of demixing matrices as a Riemannian manifold, moving away from flat Euclidean assumptions.

  1. Fisher Information Matrix (FIM) as a Metric: Instead of using a generic identity matrix for gradients, the author employs the FIM. The FIM effectively captures the intrinsic sensitivity of the likelihood function to changes in the parameters.
  2. Method of Scoring: This technique acts as a variation of Newton's method. Rather than using the actual Hessian matrix, it utilizes the expected Hessian, which corresponds to the FIM.
  3. Kronecker Product Algebra: To manage complex matrix-valued derivatives, the author utilizes Vec operators and Kronecker products. This transforms cumbersome matrix equations into vector forms that are mathematically easier to manipulate.
  4. Asymptotic Simplification: The core methodological novelty lies in analyzing the FIM at the "equilibrium" point—the state where sources are successfully separated. The author proves that at this specific point, the massive $n^2 \times n^2$ Fisher matrix becomes diagonal, allowing for a substantial reduction in computational complexity.

Key Insights and Results

The paper bridges theoretical statistics and practical algorithmic design, yielding several critical insights into the mechanics of BSS.

Inherent Equivariance

The paper proves that any learning rule following the multiplicative form $dW/dt = H(y)W$ is inherently stable concerning singularity. If the algorithm initializes with an invertible matrix, it is guaranteed to maintain invertibility. More importantly, the Method of Scoring is shown to automatically produce this desirable multiplicative structure.

  • Theorem 1 formally establishes that the maximum likelihood algorithm based on the FIM-induced natural gradient is a multiplicative updating rule that inherently possesses the equivariant property.

The "Cheap" Hessian

Inverting a Hessian matrix is typically the computational bottleneck of second-order optimization, generally requiring $O(n^6)$ operations for an $n \times n$ matrix. The author demonstrates that for BSS, the Fisher Information Matrix (the "average" Hessian) has a specific structure that makes its inverse trivial to compute at the solution point.

  • Theorem 2 provides the explicit diagonal form of the asymptotic Fisher Information Matrix ($D$). It reveals that the diagonal entries depend entirely on the moments of the source distributions (specifically $\mu_i, \lambda_i,$ and $\nu_i$).

Simplified Updating Rule

Building on the diagonal simplification, the paper presents a low-complexity updating rule (Equation 19):

$$\frac{dW}{dt} = \eta \left( \frac{1}{\mathbf{H}} \odot F(y) \right) W$$

This formulation allows for fast, efficient updates that still strictly respect the underlying statistical geometry of the problem. Furthermore, the paper demonstrates that the popular "Natural Gradient" used in Independent Component Analysis (ICA) is actually a specific approximation of this Method of Scoring, where certain statistical weights are assumed to be identity matrices.

Limitations and Future Work

While theoretically robust, the paper's derivations come with specific constraints:

  • Symmetry Assumption: The mathematical simplification achieved in Theorem 2 assumes that the probability density functions of the source signals are even functions. If the underlying sources are highly skewed, the diagonal simplification may lose accuracy.
  • Prior Knowledge Requirement: The algorithm requires an assumption about the source distributions to properly define the "score functions" ($\phi_i$). Although the author notes that the algorithm is generally robust to minor mismatches, optimal performance still relies on selecting the correct non-linearity.
  • Lack of Empirical Data: As a purely theoretical and mathematical derivation, the paper does not include empirical simulations or benchmark testing. It does not demonstrate how much faster this "Scoring" method converges compared to standard Natural Gradient descent in real-world scenarios (e.g., audio or image separation). Future work will likely need to focus on implementing Equation 19 within a practical ICA toolkit to evaluate its empirical performance.

Summary updated

Abstract

For blind source separation, when the Fisher information matrix is used as the Riemannian metric tensor for the parameter space, the steepest descent algorithm to maximize the likelihood function in this Riemannian parameter space becomes the serial updating rule with equivariant property. This algorithm can be further simplified by using the asymptotic form of the Fisher information matrix around the equilibrium.
BibTeX
@inproceedings{yang1997multiplicative,
  author = {Howard Hua Yang},
  title = {{Multiplicative Updating Rule for Blind Separation Derived from the Method of Scoring}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1997},
}

Loading related content...

Figure