Peer Review #1 of "The Modern Research Data Portal: a design pattern for networked, data-intensive science (v0.1)"

Kyle Chard, Eli Dart, Ian Foster, David Shifflett, Steven Tuecke, Jason Williams, W Allcock, J Bresnahan, R Kettimuthu, M Link, C Dumitrescu, I Raicu, Y Babuji, A Gerow, E Duede, J Basney, J Gaynor, C Borgman, M Lidman, B Mccollam, J Bryan, R Ananthakrishnan, J Pruyne, B Blaiszik, M Crosas, M Cusumano, L Rotman, B Tierney, M Hester, J Zurawski, R Dooley, M Vaughn, D Stanzione, S Terry, E Skidmore, R Egeland, T Wildishb, C.-h Huang, S Goff, S Mckay, E Lyons, A Stapleton, D Gessler, N Matasci, L Wang, M Hanlon, A Lenards, A Muir, N Merchant, S Lowry, S Mock, M Helmke, A Kubach, M Narro, N Hopkins, D Micklos, U Hilgert, M Gonzales, C Jordan, J Cazes, R Mclay, Z Lu, S Pasternak, L Koesterke, W Piel, R Grene, C Noutsos, K Gendler, X Feng, C Tang, M Lent, S.-j Kim, K Kvilekval, B Manjunath, V Tannen, A Stamatakis, M Sanderson, S Welch, K Cranston, P Soltis, D Soltis, B O'meara, C Ane, T BrutnellOpenalex Unknown 2018 · DOI · OpenAlex

Abstract: We describe best practices for providing convenient, high-speed, secure access to large data via research data portals. We capture these best practices in a new design pattern, the Modern Research Data Portal (MRDP), that disaggregates the traditional monolithic web-based data portal to achieve orders-of-magnitude increases in data transfer performance, support new deployment architectures that decouple control logic from data storage, and reduce development and operations costs. We introduce the design pattern; explain how it leverages high-performance data enclaves and cloud-based data management services; review representative examples at research laboratories and universities; and describe how to leverage Python APIs for authentication, authorization, data transfer, and data sharing.

The Bottleneck of Legacy Portals

For decades, scientific communities have relied on "Legacy Research Data Portals" (LRDPs). These are typically monolithic systems where a single web server handles every aspect of the user experience: login, metadata searching, and the actual serving of files. While functional for small datasets, this model has reached a breaking point as scientific data scales from gigabytes to petabytes.

Legacy Research Data Portal (LRDP) architecture where a monolithic server handles all functions behind an enterprise firewall.

LRDPs suffer from three primary bottlenecks. First, performance is limited because standard web servers sitting behind restrictive enterprise firewalls are not optimized for massive data movement. Second, reliability is poor; simple "click-to-download" browser links frequently fail during multi-terabyte transfers without mechanisms for verification or resumption. Finally, operational complexity forces research teams to become experts in security and high-performance networking, leading to expensive, siloed, and often insecure implementations.

The MRDP Design Pattern

The Modern Research Data Portal (MRDP) addresses these issues by decomposing the portal into three distinct, specialized components. This disaggregated architecture separates the "control plane" (the user interface and logic) from the "data plane" (the actual movement of bits).

The Modern Research Data Portal (MRDP) architecture featuring a Science DMZ and separated control and data paths.

1. The Portal Server

This is a lightweight web application (often built with Python/Flask) that manages the control channel. It handles searching, browsing metadata, and mapping users to specific datasets. Crucially, the portal server never actually touches the data files; it only orchestrates their movement.

2. The Science DMZ and DTNs

Data is stored in a specialized network enclave known as a Science DMZ. This enclave contains Data Transfer Nodes (DTNs)—high-performance servers optimized specifically for moving data. By sitting outside the restrictive enterprise firewall but protected by specialized security policies, DTNs can leverage the full bandwidth of high-speed research networks.

3. Cloud-Based Management

The MRDP pattern "outsources" the heavy lifting of data management to external cloud services, specifically Globus, via REST APIs. This includes:

  • Authentication: Using OAuth2 and OpenID Connect to allow users to log in with existing institutional identities.
  • Transfer: Utilizing GridFTP for high-speed, reliable, third-party data movement that can be monitored and managed via the cloud.
  • Sharing: Managing granular access permissions without requiring the creation of local system accounts for every user.

Real-World Performance and Scalability

The efficacy of the MRDP pattern is demonstrated through several high-impact deployments. For instance, the NCAR Research Data Archive (RDA) serves over 24,000 users and moves approximately 1.9 Petabytes of data annually. Transfers from NCAR to the NERSC supercomputing center have achieved sustained rates exceeding 1 GB/s (8 Gbps) over a 10 Gbps connection.

Other notable implementations include Argonne’s Petrel, which has recorded moving over 1 Petabyte of data in a single day, and the Sanger Imputation Service for genomic data. These examples show that by separating control logic from data storage, portals can remain responsive and secure even while massive datasets are transferred in the background.

Conclusion and Future Directions

The MRDP design pattern significantly reduces the technical debt of research teams while providing users with a high-performance, reliable experience. By leveraging cloud services and specialized network architectures, researchers can focus on science rather than infrastructure. Future work involves extending this pattern to include "on-demand" computation, where portals trigger containerized analysis (using Docker or Singularity) on data before it is delivered to the end user.

Summary updated

Abstract

We describe best practices for providing convenient, high-speed, secure access to large data via research data portals.We capture these best practices in a new design pattern, the Modern Research Data Portal, that disaggregates the traditional monolithic web-based data portal to achieve orders-of-magnitude increases in data transfer performance, support new deployment architectures that decouple control logic from data storage, and reduce development and operations costs.We introduce the design pattern; explain how it leverages high-performance data enclaves and cloud-based data management services;review representative examples at research laboratories and universities, including both experimental facilities and supercomputer sites; describe how to leverage Python APIs for authentication, authorization, data transfer, and data sharing; and use coding examples to demonstrate how these APIs can be used to implement a range of research data portal capabilities.Sample code at a companion web site, https://docs.globus.org/mrdp,provides application skeletons that readers can adapt to realize their own research data portals.
BibTeX
@inproceedings{chard2018peer,
  author = {Kyle Chard and Eli Dart and Ian Foster and David Shifflett and Steven Tuecke and Jason Williams and W Allcock and J Bresnahan and R Kettimuthu and M Link and C Dumitrescu and I Raicu and I Foster and Y Babuji and K Chard and A Gerow and E Duede and J Basney and J Gaynor and C Borgman and K Chard and M Lidman and B Mccollam and J Bryan and R Ananthakrishnan and S Tuecke and I Foster and K Chard and J Pruyne and B Blaiszik and R Ananthakrishnan and S Tuecke and I Foster and K Chard and S Tuecke and I Foster and M Crosas and M Cusumano and E Dart and L Rotman and B Tierney and M Hester and J Zurawski and R Dooley and M Vaughn and D Stanzione and S Terry and E Skidmore and R Egeland and T Wildishb and C.-H Huang and S Goff and M Vaughn and S Mckay and E Lyons and A Stapleton and D Gessler and N Matasci and L Wang and M Hanlon and A Lenards and A Muir and N Merchant and S Lowry and S Mock and M Helmke and A Kubach and M Narro and N Hopkins and D Micklos and U Hilgert and M Gonzales and C Jordan and E Skidmore and R Dooley and J Cazes and R Mclay and Z Lu and S Pasternak and L Koesterke and W Piel and R Grene and C Noutsos and K Gendler and X Feng and C Tang and M Lent and S.-J Kim and K Kvilekval and B Manjunath and V Tannen and A Stamatakis and M Sanderson and S Welch and K Cranston and P Soltis and D Soltis and B O'meara and C Ane and T Brutnell and Ian T. Foster},
  title = {{Peer Review \#1 of "The Modern Research Data Portal: a design pattern for networked, data-intensive science (v0.1)"}},
  booktitle = {Openalex Unknown},
  year = {2018},
  doi = {10.7287/peerj-cs.144v0.1/reviews/1},
}
Figure