Cluster, Classify, Regress: A General Method For Learning Discountinous Functions

David E. Bernholdt, Mark R. Cianciosa, Clement Etienam, David L. Green, Kody J. H. Law, Jin Myung ParkarXiv 2019 · OpenReview

TL;DR

This paper introduces a novel "Cluster, Classify, Regress" (CCR) framework to tackle supervised learning problems with highly nonlinear and discontinuous functions. By breaking down the problem into three stages, CCR effectively handles complex data patterns that often stump traditional machine learning methods, offering a powerful new tool for scientific and engineering applications.

What Problem Does This Paper Address?

Many real-world problems in science and engineering involve functions that are not smooth and continuous. Think of phenomena with sharp transitions or abrupt changes, like phase transitions in physics or sudden shifts in biological systems. Standard machine learning techniques, particularly regression models, struggle with these discontinuities, often failing to accurately capture the underlying behavior. While methods exist to handle these issues, a unified and general approach that effectively partitions the problem space based on these discontinuities has been lacking. This paper aims to fill that gap by proposing a general framework that can leverage existing machine learning tools to reconstruct these complex, discontinuous functions.

Methods

The core of the paper is the "Cluster, Classify, Regress" (CCR) method, a three-stage pipeline:

  1. Cluster: The algorithm first clusters the input-output data pairs. This step is crucial for identifying regions of the data that behave more smoothly. The clustering aims to assign a "label" to each data point, effectively partitioning the dataset based on inherent structure, not just input features. K-means clustering is used as an example, with the number of clusters (L) determined by the elbow method.
  2. Classify: Next, a classifier is trained to predict the cluster label for new, unseen input data. This means that for any given input, the classifier will assign it to one of the previously identified clusters. The output data (y) is ignored in this phase, focusing solely on mapping inputs to their corresponding cluster labels.
  3. Regress: Finally, for each identified cluster, a separate regression model is trained. This regression model only uses the data points that were assigned to that specific cluster. This allows each regression model to focus on a "smoother" subset of the data, avoiding the challenges posed by discontinuities. The final prediction for an input is then made by first classifying it into a cluster and then using the corresponding regression model.

This approach can be viewed as a form of "deep learning," where each stage can potentially be a complex model itself, and the entire pipeline can be nested.

Datasets

The paper evaluates the proposed CCR method on several synthetic "toy" problems and one real-world example from plasma fusion simulation.

  • Synthetic Examples: The paper presents numerical experiments with functions like $f_1(x) = x \mathbb{1}_{x \ge 1}$, $f_2(x) = (x+1)\mathbb{1}_{x<0} + x\mathbb{1}_{x>0}$, $f_3(x)$ (a piecewise function with exponential and step components), and $f_4(x) = f_3(x_1)f_3(x_2)$ (a 2D version of $f_3$). These examples are designed to test the method's ability to handle discontinuities and nonlinearities.
  • Tokamak Plasma Transport: A real-world example involves modeling turbulent ion energy transport in tokamak plasmas using an analytic stiff transport model. This model is known to exhibit highly nonlinear and discontinuous behavior based on input gradients.

Key Insights and Ideas

  • Decomposition is Key: The fundamental insight is that by decomposing a complex, discontinuous function into simpler, continuous pieces, the problem becomes tractable for standard regression techniques.
  • Input-Output Clustering: Clustering based on both input and output data points (rather than just inputs) is crucial for identifying regions that are smooth in their input-output relationship, even if the overall function is discontinuous.
  • "Super Deep Learning": The CCR framework can be seen as a form of "super deep learning" because it stacks multiple machine learning components (clustering, classification, regression) in a hierarchical manner, and each component can itself be a complex deep learning architecture.
  • Robustness to Discontinuities: The method demonstrates robustness by effectively handling sharp transitions, which are often problematic for other approaches. The use of multiple clusters, even if some initially span discontinuities, helps to isolate smoother regions.

Key Results

The paper demonstrates the effectiveness of the CCR method through several numerical examples:

  • Accurate Reconstruction: The CCR method successfully reconstructs the tested discontinuous functions, achieving high accuracy. For instance, in numerical example 5, the L2 and R2 scores are consistently high across different configurations, with L2 scores often exceeding 0.99.
  • Handling Discontinuities: The method is shown to recover discontinuities in functions like $f_2(x)$ and $f_3(x)$, outperforming standard MLPs and DNNs in these cases.
  • Tokamak Application: The method is applied to a complex, high-dimensional problem in fusion energy research, showing promising results in modeling turbulent transport.
  • Active Learning Extension: The paper also proposes active learning strategies that can efficiently select informative data points to label, reducing the cost of data acquisition while maintaining high accuracy. Table 2 shows that active learning can achieve comparable error rates with significantly fewer labeled points compared to passive learning.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that the regression stage can be computationally expensive if there are many clusters, as it requires training a separate regressor for each. However, this stage is embarrassingly parallel.
  • Struggle with Vanishing/Appearing Discontinuities: Remark 2.3 points out that the method might struggle if discontinuities appear or vanish within the domain, though this can often be mitigated by using a larger number of clusters.
  • Bayesian Formulation: The authors suggest a Bayesian formulation of the CCR method as a direction for future work, which could provide a more principled way to handle uncertainty and model selection.
  • Generalization to Other ML Components: While K-means, MLPs, and decision trees are used as examples, the framework is flexible and can incorporate any combination of clustering, classification, and regression algorithms, including more advanced deep learning architectures.

Summary updated

Abstract

This paper presents a method for solving the supervised learning problem in which the output is highly nonlinear and discontinuous. It is proposed to solve this problem in three stages: (i) cluster the pairs of input-output data points, resulting in a label for each point; (ii) classify the data, where the corresponding label is the output; and finally (iii) perform one separate regression for each class, where the training data corresponds to the subset of the original input-output pairs which have that label according to the classifier. It has not yet been proposed to combine these 3 fundamental building blocks of machine learning in this simple and powerful fashion. This can be viewed as a form of deep learning, where any of the intermediate layers can itself be deep. The utility and robustness of the methodology is illustrated on some toy problems, including one example problem arising from simulation of plasma fusion in a tokamak.
BibTeX
@misc{bernholdt2019cluster,
  author = {David E. Bernholdt and Mark R. Cianciosa and Clement Etienam and David L. Green and Kody J. H. Law and Jin Myung Park},
  title = {{Cluster, Classify, Regress: A General Method For Learning Discountinous Functions}},
  year = {2019},
  eprint = {1905.06220},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.06220},
}
Figure