Apprentissage sur Données Massives; trois cas d'usage avec R, Python et Spark.

Philippe Besse, Brendan Guillouet, Jean–Michel LoubesHAL 2017

TL;DR

This paper compares the performance of R, Python (Scikit-learn), and Spark (MLlib) for big data machine learning tasks across three use cases: character recognition, movie recommendation, and product categorization. The key takeaway is that while Spark excels at data preparation and collaborative filtering, traditional methods in R and Python often outperform Spark's implementations for standard learning algorithms, especially when data fits within a single machine's memory.

What Problem Does This Paper Address?

The paper addresses the growing challenge of choosing the right tools and environments for analyzing massive datasets. As data volumes explode, traditional statistical and machine learning approaches, often implemented in environments like R, face limitations. This necessitates exploring distributed computing frameworks like Spark. However, the landscape of big data technologies is complex and rapidly evolving, making it difficult for statisticians and data scientists to navigate. The authors aim to provide a practical comparison of popular environments (R, Python, Spark) on real-world tasks to guide these choices, focusing on the trade-offs between integrated (single-machine) and distributed architectures.

Methods

The paper compares three primary environments:

  • R: A well-established statistical computing environment.
  • Python (Scikit-learn): A popular general-purpose programming language with a robust machine learning library.
  • Spark (MLlib): A distributed computing framework designed for large-scale data processing and machine learning.

The authors evaluate these environments across three distinct use cases:

  1. Character Recognition (MNIST): A classic supervised learning problem.
  2. Movie Recommendation (MovieLens): A collaborative filtering and matrix completion task.
  3. Product Categorization (Cdiscount): A text classification problem involving large text datasets.

For each use case, they analyze performance in terms of execution time and prediction accuracy. They also investigate the impact of data size and, in the case of Spark, the number of executors.

Datasets

  • MNIST: A dataset of handwritten digits, commonly used for benchmarking classification algorithms.
  • MovieLens: Datasets of user ratings for movies, available in various sizes (100k, 1M, 10M, 20M ratings), used for recommendation system evaluation.
  • Cdiscount: A large dataset of product descriptions and categories from an e-commerce platform (3.5 GB, 15 million lines), used for text classification.

Key Insights and Ideas

  • Spark's Strength in Data Preparation: Spark demonstrates significant advantages in data cleaning, transformation, and preparation (data munging) for large datasets, outperforming integrated architectures in terms of speed.
  • Integrated vs. Distributed Trade-offs: For many standard machine learning algorithms (e.g., random forests, logistic regression), when data fits into memory, integrated architectures (like a powerful single machine running R or Python) often provide better or comparable prediction accuracy with faster execution times than distributed Spark implementations.
  • Collaborative Filtering Advantage for Spark: Spark shows strong performance in collaborative filtering tasks, particularly for matrix factorization, which is crucial for recommendation systems.
  • Memory Constraints in Distributed Systems: Spark's distributed nature can introduce memory management challenges, especially for complex models, potentially leading to errors or requiring careful tuning of cluster resources. This contrasts with integrated systems that can leverage virtual memory (swapping) at the cost of speed.
  • Scalability Nuances: While Spark is designed for scalability, the actual performance gains can vary significantly depending on the algorithm and the specific task. Linear scalability is not always achieved, and sometimes an integrated architecture with sufficient memory can be more efficient.
  • R and Python Remain Viable: For many tasks, especially those requiring deep understanding, interpretation, and less extreme data volumes, R and Python (Scikit-learn) remain highly effective and often more performant than distributed solutions.

Key Results

  • Character Recognition: R (ranger) and Python (Scikit-learn) achieved comparable accuracy. Spark MLlib's random forest implementation faced memory constraints, limiting the number of trees and depth, leading to less competitive results.
  • Movie Recommendation: Spark MLlib's NMF (Non-negative Matrix Factorization) using ALS (Alternating Least Squares) provided satisfactory results (RMSE of 0.82) for matrix completion, outperforming R's softImpute.
  • Product Categorization: For text cleaning, Spark showed better scalability than Python. However, for vectorization (TF-IDF) and learning (logistic regression), integrated architectures with sufficient memory were generally faster and more precise, especially as the number of variables (n_hash) increased. Spark MLlib struggled with memory for larger n_hash values.

Limitations and Future Work

  • Limited Scope: The study focuses on a subset of algorithms and environments. More advanced deep learning algorithms and other distributed frameworks are not covered.
  • Maturity of Technologies: The paper acknowledges that Spark and Scikit-learn are rapidly evolving, and findings might change with newer versions.
  • Data Size vs. Accuracy: For character recognition, the paper notes that beyond a certain data size, accuracy does not improve significantly, suggesting that data quantity is not always the primary driver of performance.
  • Future Work: The authors suggest that for future work, exploring techniques like sub-sampling without replacement for random forests could be beneficial. They also highlight the need for more advanced modeling techniques that possess invariance properties for tasks like character recognition. The paper implicitly calls for continued benchmarking as technologies mature.

Summary updated

Abstract

International audience
BibTeX
@inproceedings{besse2017apprentissage,
  author = {Philippe Besse and Brendan Guillouet and Jean–Michel Loubes},
  title = {{Apprentissage sur Données Massives; trois cas d'usage avec R, Python et Spark.}},
  booktitle = {HAL},
  year = {2017},
}
Figure