Neural network modeling of data with gaps: method of principal curves Carleman's formula, and other

Alexander N. Gorban, A. A. Rossiev, Donald C. Wunsch IiarXiv 2003 · arXiv

Abstract

This paper presents a robust framework for the reconstruction of incomplete datasets and the correction of erroneous values using a hierarchy of manifold-based models. By transitioning from linear principal components to complex Self-Organizing Curves (SOC) and integrating Carleman’s formula for analytical continuation, the authors develop a "neural conveyor" architecture. This system is capable of data recovery even when up to 50% of values are missing, demonstrating high efficacy in both historical political analysis and complex biomedical datasets.

Introduction

In practical data science, datasets are rarely pristine. Missing values (gaps) and erroneous entries (noise) are ubiquitous, often rendering standard regression techniques ineffective—particularly when gaps are irregularly distributed or when the feature count approaches the number of observations. This research addresses the challenge of "plausible" data recovery. Rather than seeking a singular statistical "truth," the proposed methods identify the underlying manifold structure of the data, using "hidden curves" to bridge gaps and repair anomalies.

The Neural Conveyor Architecture

A central innovation of this work is the "Neural Conveyor." Unlike traditional feed-forward neural networks, this architecture consists of a sequence of modular units. Each "neuron" in the conveyor is responsible for extracting a single principal factor—whether linear or non-linear—and passing the residual variance to the subsequent unit. This pipelined approach ensures high interpretability and allows for efficient parallel processing of data streams.

Methodological Framework

The authors propose a three-tiered hierarchy of models to handle increasing levels of data complexity:

1. Linear Generalization

The foundation is a generalization of the Principal Component Method (PCM) adapted for incomplete tables. Traditional PCM requires data centering, which is impossible when the mean is unknown due to gaps. The authors bypass this by using an iterative singular expansion to approximate the data matrix as a sum of rank-1 matrices, effectively learning the mean and components simultaneously.

2. Quasilinear Models and Carleman’s Formula

To move beyond simple linearity, the authors construct "principal curves." A significant contribution here is the application of Carleman’s formula—a tool from complex analysis typically used for analytical continuation. This allows the model to perform extrapolation and interpolation with greater stability and smoothness than standard polynomial methods, especially at the "edges" of the dataset.

3. Self-Organizing Curves (SOC)

The most advanced tier involves essentially non-linear modeling through Self-Organizing Curves. Inspired by Kohonen’s Self-Organizing Maps (SOM), the SOC method treats the modeling manifold as an elastic beam. The model minimizes a functional balancing three critical measures:

  • Approximation: How closely the curve fits the known data.
  • Connectivity: Maintaining proximity between adjacent points on the curve.
  • Nonlinearity (Elasticity): Resistance to bending, which acts as a regularizer to prevent overfitting to noise.

Experimental Results

The framework was validated across two diverse domains:

Historical Political Analysis

Using a dataset of US Presidential Elections (1860–1980) defined by 12 binary features, the models successfully predicted the outcomes of several subsequent elections, including the 1984 Reagan victory, the 1988 Bush election, and both of Bill Clinton’s terms.

Biomedical Data Repair

In a study of 1,700 patients with Myocardial Infarction complications (126 parameters), the SOC model demonstrated a unique "repair" capability. For instance, if a patient's recorded age was inconsistent with their biological profile (as suggested by the other 125 parameters), the model could suggest a more "plausible" medical-biological age.

Performance and Resilience

The linear and quasilinear models remained effective with up to 25% missing data. The SOC-based models pushed this threshold further, successfully recovering data with up to 50% of values missing. Furthermore, the SOC model consistently achieved higher approximation percentages (often exceeding 95%) with fewer factors compared to traditional linear approaches.

Discussion and Limitations

While powerful, the method prioritizes "plausibility" over formal statistical provability. The lack of traditional confidence intervals may be a limitation in high-stakes environments. Additionally, the SOC method involves non-convex optimization, which can be computationally intensive and susceptible to local minima, though the authors suggest "annealing" techniques as a mitigation strategy. Finally, the quality of the manifold is sensitive to the initial normalization of the data.

Conclusion

The "neural conveyor" and the use of principal curves provide a sophisticated alternative to standard imputation methods. By leveraging geometric manifolds and complex analysis, this framework offers a robust solution for modeling "dirty" data in complex, high-dimensional spaces.

Summary updated

Abstract

A method of modeling data with gaps by a sequence of curves has been developed. The new method is a generalization of iterative construction of singular expansion of matrices with gaps. Under discussion are three versions of the method featuring clear physical interpretation: linear - modeling the data by a sequence of linear manifolds of small dimension; quasilinear - constructing "principal curves: (or "principal surfaces"), univalently projected on the linear principal components; essentially non-linear - based on constructing "principal curves": (principal strings and beams) employing the variation principle; the iteration implementation of this method is close to Kohonen self-organizing maps. The derived dependencies are extrapolated by Carleman's formulas. The method is interpreted as a construction of neural network conveyor designed to solve the following problems: to fill gaps in data; to repair data - to correct initial data values in such a way as to make the constructed models work best; to construct a calculator to fill gaps in the data line fed to the input.
BibTeX
@misc{gorban2003neural,
  author = {Alexander N. Gorban and A. A. Rossiev and Donald C. Wunsch II},
  title = {{Neural network modeling of data with gaps: method of principal curves Carleman's formula, and other}},
  year = {2003},
  eprint = {cond-mat/0305508},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/cond-mat/0305508},
}
Figure