Introduction
In the late 1980s, connectionist models (neural networks) almost exclusively relied on gradient descent to minimize a least-squares error function—the ubiquitous Mean Squared Error (MSE). While highly effective, the least-squares approach is mathematically tied to the assumption that noise in the training data follows a Gaussian distribution.
However, real-world data is rarely that well-behaved. Outliers can disproportionately pull the gradient, and the geometric "shape" of the error surface in a standard Euclidean space might not be the most efficient for every task. This paper addresses this fundamental limitation by asking: What happens if we change the very definition of "error distance" during training? By generalizing the standard back-propagation algorithm to use the Minkowski-r power metric, the authors demonstrate that treating the error power ($r$) as a tunable parameter allows models to become significantly more robust to noise or better at finding simple, compact decision boundaries.
The Minkowski-r Method
The core contribution of the paper is Minkowski-r Back-Propagation. Instead of the standard Euclidean error $E = \frac{1}{2} \sum (y_i - \hat{y}_i)^2$, the authors substitute the generalized Minkowski metric:
$$E = \frac{1}{r} \sum |y_i - \hat{y}_i|^r$$
To make this work in practice, the authors derived the generalized partial derivatives required to update network weights for any value of $r$. Furthermore, they proposed an adaptive method that defines a gradient over $r$ itself. This allows the network to perform maximum likelihood estimation to dynamically find the "optimal" metric for a given dataset during training. Because $r$ dictates the fundamental shape of the loss landscape, the authors note a crucial implementation heuristic: if $r$ is being learned, it must be updated much more slowly than the network weights (e.g., once every 10 weight updates) to maintain training stability.
Key Insights and Experimental Results
The authors evaluated their approach on a mix of synthetic and real-world-inspired tasks, including 3D surface recovery, the classic XOR problem, Minsky and Papert's "Mesh" classification problem, and a speech recognition task. Modifying $r$ yielded distinct advantages depending on the nature of the data.
Robustness to Outliers ($r < 2$)
When $r$ is small (approaching the "city-block" or $L_1$ metric), the network gives less weight to large deviations. This transforms the network into a "robust estimator" capable of ignoring noisy outliers that would otherwise distort the model's internal representation.
This was vividly demonstrated in a synthetic 3D surface recovery task. When tasked with recovering a smooth Gaussian-based surface corrupted with heavy uniform random noise, a network using $r=1.2$ successfully recovered the original smooth shape. In contrast, the standard $r=2$ (MSE) approach produced significant distortions at the peaks and bases, as it over-penalized the extreme noise values.
Simplifying Decision Boundaries ($r > 2$)
Conversely, larger values of $r$ weight large deviations more heavily. In complex classification tasks like "The Mesh"—which involves two meshed clusters requiring specific hyper-plane partitions—higher $r$ values force the network to pull decision boundaries closer to the targets.
This often results in a more parsimonious solution. For example, a network with 16 hidden units using $r=2$ typically found a solution using five decision boundaries. By simply increasing the metric to $r=3$, the network converged on a simpler, more optimal solution using only three boundaries.
Small Sample Advantage in Speech
The paper also highlights that $r=2$ is only optimal if the underlying conditional probability distributions are Gaussian. For a speech recognition dataset of 50 spoken digits, higher $r$ values (e.g., $r=3$) showed substantially better generalization when training sets were very small. On a task with only 10 training samples, the $r=3$ model achieved roughly 80% accuracy, compared to a mere 15% for the standard $r=2$ model. The authors suggest the higher $r$ metric better captured the inherent compactness of the data clusters in this domain.
Limitations and Trade-offs
While powerful, Minkowski-r Back-Propagation comes with trade-offs:
- Computational Cost and Convergence: Convergence time tends to increase roughly linearly as $r$ increases. Higher $r$ values make the learning process slower, which could bottleneck very large-scale networks.
- Local Minima: While dynamically learning $r$ is theoretically appealing, seeking an optimal $r$ in large, complex problems can lead the network into local minima.
- Implementation Complexity: The Minkowski-r gradient is non-linear and more complex to implement than standard MSE, requiring careful programmatic handling of the signum function and absolute values.
- Metric Axioms: The authors note that $r < 1$ values are technically non-metric (they violate the triangle inequality), though they remain an unexplored mathematical curiosity within this framework.
Ultimately, this research highlights that the default choice of Mean Squared Error is exactly that—a choice. By treating the error metric as a flexible component of the architecture, practitioners can tailor their networks to be more robust to noise or more aggressive in finding compact representations.
Summary updated