Transformative Machine Learning

Iván Olier, Oghenejokpeme I. Orhobor, Joaquin Vanschoren, Ross D. KingarXiv 2018 · arXiv

Abstract

Transformative Machine Learning (TML) is a novel framework designed to overcome the limitations of standard machine learning and deep learning in data-scarce scientific domains. By converting raw intrinsic features into extrinsic representations—defined as the predictions of models trained on related tasks—TML achieves significant performance gains while maintaining model interpretability.

Introduction

In scientific fields such as drug discovery and genomics, researchers frequently encounter "small data" problems where the number of available samples is limited. While deep neural networks (DNNs) are powerful, they typically require vast datasets to learn effective internal representations. Furthermore, the "black-box" nature of DNNs often conflicts with the scientific requirement for explainable models.

Transformative Machine Learning (TML) addresses these challenges by exploiting the inherent relationships between numerous related tasks. Instead of relying solely on hand-crafted "intrinsic" features, TML uses the outputs of models trained on related problems as "extrinsic" features, effectively transferring knowledge across the task space.

The TML Framework

The TML process is structured as a two-stage learning pipeline:

Stage 1: Intrinsic Learning

In the first stage, separate predictive models are trained for each of $n$ related tasks. These models utilize standard, domain-specific intrinsic descriptors (e.g., molecular weight in chemistry or gene expression levels in biology).

The first stage of TML involves training individual models on intrinsic features across multiple related tasks.

Stage 2: Extrinsic Transformation

To solve a specific target task, the input data is passed through the models developed in Stage 1 for the other $n-1$ tasks. The resulting predictions serve as a new set of extrinsic features. A final model is then trained for the target task using these extrinsic descriptors.

In the TML process, a target task (e.g., rabbit classification) is solved by using predictions from models trained on related tasks as new features.

Unlike the implicit representations learned by deep learning, TML's extrinsic features are explicit and numeric, representing how a particular sample relates to a library of known problems.

Experimental Evaluation

The authors validated TML across three distinct scientific domains:

  • Quantitative Structure-Activity Relationship (QSAR): Using 2,219 drug targets from the ChEMBL database, TML achieved a >10% mean improvement in Root Mean Squared Error (RMSE) over standard Random Forests and SVMs.
  • Gene Expression: On the LINCS Phase II dataset (978 landmark genes), TML improved prediction accuracy by 4% to 7% across nearly 1,000 models.
  • Meta-Learning: Predicting algorithm performance on OpenML tasks saw the most significant gain, with a >55% mean improvement compared to using standard dataset descriptors.

Discussion and Key Insights

The success of TML is rooted in several key observations:

  • Superiority of Extrinsic Features: Relational information (e.g., "this molecule behaves like a known inhibitor") is often more predictive than raw physical properties.
  • The Role of Non-linearity: TML benefits significantly from non-linear base models (e.g., Random Forests), which capture complex relationships that linear models miss.
  • Inherent Explainability: Because features correspond to specific related tasks, researchers can trace predictions back to well-understood biological or chemical contexts.
  • Dynamic Task Similarity: TML does not assume task similarity; the final model learns which extrinsic features are most relevant for the target task.

Conclusion

Transformative Machine Learning provides a robust, data-efficient, and explainable alternative to traditional machine learning and deep learning for scientific discovery. While computationally more intensive than single-task learning, its ability to extract high-value representations from small datasets makes it a powerful tool for modern research. Future work into "Second-Order TML" may further amplify these performance gains.

Summary updated

Abstract

The key to success in machine learning (ML) is the use of effective data representations. Traditionally, data representations were hand-crafted. Recently it has been demonstrated that, given sufficient data, deep neural networks can learn effective implicit representations from simple input representations. However, for most scientific problems, the use of deep learning is not appropriate as the amount of available data is limited, and/or the output models must be explainable. Nevertheless, many scientific problems do have significant amounts of data available on related tasks, which makes them amenable to multi-task learning, i.e. learning many related problems simultaneously. Here we propose a novel and general representation learning approach for multi-task learning that works successfully with small amounts of data. The fundamental new idea is to transform an input intrinsic data representation (i.e., handcrafted features), to an extrinsic representation based on what a pre-trained set of models predict about the examples. This transformation has the dual advantages of producing significantly more accurate predictions, and providing explainable models. To demonstrate the utility of this transformative learning approach, we have applied it to three real-world scientific problems: drug-design (quantitative structure activity relationship learning), predicting human gene expression (across different tissue types and drug treatments), and meta-learning for machine learning (predicting which machine learning methods work best for a given problem). In all three problems, transformative machine learning significantly outperforms the best intrinsic representation.
BibTeX
@misc{olier2018transformative,
  author = {Iván Olier and Oghenejokpeme I. Orhobor and Joaquin Vanschoren and Ross D. King},
  title = {{Transformative Machine Learning}},
  year = {2018},
  eprint = {1811.03392},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.03392},
}
Figure