In-transit molecular dynamics analysis with Apache flink

Henrique C. Zanúz, Bruno Raffin, Omar A. Mures, Emilio J. PadrónISAV@SC 2018 · DOI

TL;DR

This paper introduces a novel framework for performing real-time analysis of molecular dynamics (MD) simulations directly as they run, using Apache Flink. This "in-transit" approach bypasses traditional post-hoc analysis bottlenecks, enabling scientists to gain insights faster and more efficiently by leveraging Big Data technologies within High-Performance Computing (HPC) environments.

What Problem Does This Paper Address?

Modern scientific simulations, particularly in fields like molecular dynamics (MD), are generating an unprecedented amount of data. This deluge of information poses a significant challenge for traditional analysis pipelines, which typically involve storing all raw simulation data and then processing it later (post-hoc analysis). The I/O (input/output) systems in HPC environments often struggle to keep pace with the data generation rates, leading to storage and analysis bottlenecks. Furthermore, developing and deploying custom parallel analysis code for these simulations can be complex and time-consuming for scientists, requiring expertise in low-level HPC programming models like MPI and OpenMP. This paper addresses the need for a simplified, scalable, and efficient way to perform analysis concurrently with the simulation, reducing the reliance on cumbersome post-hoc methods.

Methods

The core of the proposed solution is the integration of Apache Flink, a powerful Big Data stream processing engine, into an HPC workflow. The framework connects a parallel MD simulation (using the CoMD mini-app) to Flink workers via ZeroMQ for real-time data streaming. Flink then processes this data in-memory using a window-based map/reduce model, taking care of deployment, load balancing, and fault tolerance. The analyzed results, along with the raw simulation data, are then stored in Apache HBase, a scalable NoSQL database.

Key methodological aspects include:

  • In-Transit Data Processing: Instead of saving all raw data, Flink analyzes it as it's generated, reducing the I/O burden.
  • Apache Flink Integration: Leveraging Flink's high-level programming model and its robust runtime for distributed stream processing.
  • ZeroMQ for Communication: Used as an asynchronous message library to efficiently stream data from the simulation to Flink.
  • Apache HBase for Storage: A scalable NoSQL database to store both raw simulation data and analysis results, enabling both in-transit and subsequent post-hoc analysis.
  • Two MD Analytics Kernels: The paper demonstrates the framework's expressivity by implementing two common MD analysis tasks: a position histogram of atoms and an atom neighbor identification based on a cutoff distance. These kernels utilize Morton indexing (z-order curve) for efficient space partitioning and neighbor searching.

Datasets

The paper does not introduce new benchmark datasets. Instead, it uses data generated by the CoMD (Classical molecular dynamics proxy application) mini-app, which simulates molecular dynamics workloads. The size of the generated data varies, with experiments involving up to 32 million atoms (63 GB of raw data) and scaled-up versions reaching 252 GB.

Key Insights and Ideas

  • Big Data Meets HPC: The paper successfully demonstrates that Big Data frameworks like Apache Flink can be effectively integrated into HPC environments for scientific data analysis, offering a more accessible programming model than traditional HPC approaches.
  • Bypassing I/O Bottlenecks: By processing data "in-transit," the framework significantly alleviates the pressure on traditional storage systems, which are often a bottleneck in large-scale simulations.
  • Simplified Parallel Analytics: Flink's high-level abstractions abstract away much of the complexity of parallel programming, load balancing, and fault tolerance, making it easier for domain scientists to implement custom analyses.
  • Unified Analysis Workflow: The framework supports both in-transit analysis and traditional post-hoc analysis, as raw data is also stored. The same analysis kernels can be reused for both.
  • Trade-off Between Performance and Usability: The authors highlight that scientists may be willing to trade some raw performance for a significant reduction in development and deployment complexity, a trade-off this framework facilitates.

Key Results

The framework achieved a sustainable throughput of 71 MB/s (or 8.875 MB/s per node) for direct data injection and histogram computation with 125 CoMD processes. In a scaled-up experiment with 512 CoMD processes, the throughput reached 100 MB/s. This allowed for the storage of 63 GB of raw data (and computed histograms) within minutes.

However, the neighbor identification analysis showed a significantly lower throughput of 1.10 MB/s, attributed to data replication and shuffling overheads in Flink. The weak scalability test also indicated that while throughput increased with scale, it did not scale linearly, suggesting room for optimization.

Limitations and Future Work

The authors acknowledge several limitations and propose future directions:

  • Neighbor Identification Performance: The current implementation of neighbor identification is a bottleneck due to data duplication and shuffling in Flink. Future work will focus on exploring alternative algorithms and Flink operators to optimize this.
  • Storage System Choice: While HBase was chosen for its Big Data relevance, other systems like Cassandra might offer better write performance.
  • Hardware: Experiments were conducted on a specific cluster (Grid 5000). Further testing on systems with InfiniBand networks and SSDs is planned.
  • Software Availability: The framework was not publicly available at the time of publication but was planned to be open-sourced.
  • Refining Analytics Algorithms: Continuous improvement of the analytics kernels and exploration of different Flink operators for performance gains.

Summary updated

BibTeX
@inproceedings{zanuz2018transit,
  author = {Henrique C. Zanúz and Bruno Raffin and Omar A. Mures and Emilio J. Padrón},
  title = {{In-transit molecular dynamics analysis with Apache flink}},
  booktitle = {ISAV@SC},
  year = {2018},
  doi = {10.1145/3281464.3281469},
}
Figure