Abstract: We describe best practices for providing convenient, high-speed, secure access to large data via research data portals. We capture these best practices in a new design pattern, the Modern Research Data Portal (MRDP), that disaggregates the traditional monolithic web-based data portal to achieve orders-of-magnitude increases in data transfer performance, support new deployment architectures that decouple control logic from data storage, and reduce development and operations costs. We introduce the design pattern; explain how it leverages high-performance data enclaves and cloud-based data management services; review representative examples at research laboratories and universities; and describe how to leverage Python APIs for authentication, authorization, data transfer, and data sharing.
The Bottleneck of Legacy Portals
For decades, scientific communities have relied on "Legacy Research Data Portals" (LRDPs). These are typically monolithic systems where a single web server handles every aspect of the user experience: login, metadata searching, and the actual serving of files. While functional for small datasets, this model has reached a breaking point as scientific data scales from gigabytes to petabytes.
LRDPs suffer from three primary bottlenecks. First, performance is limited because standard web servers sitting behind restrictive enterprise firewalls are not optimized for massive data movement. Second, reliability is poor; simple "click-to-download" browser links frequently fail during multi-terabyte transfers without mechanisms for verification or resumption. Finally, operational complexity forces research teams to become experts in security and high-performance networking, leading to expensive, siloed, and often insecure implementations.
The MRDP Design Pattern
The Modern Research Data Portal (MRDP) addresses these issues by decomposing the portal into three distinct, specialized components. This disaggregated architecture separates the "control plane" (the user interface and logic) from the "data plane" (the actual movement of bits).
1. The Portal Server
This is a lightweight web application (often built with Python/Flask) that manages the control channel. It handles searching, browsing metadata, and mapping users to specific datasets. Crucially, the portal server never actually touches the data files; it only orchestrates their movement.
2. The Science DMZ and DTNs
Data is stored in a specialized network enclave known as a Science DMZ. This enclave contains Data Transfer Nodes (DTNs)—high-performance servers optimized specifically for moving data. By sitting outside the restrictive enterprise firewall but protected by specialized security policies, DTNs can leverage the full bandwidth of high-speed research networks.
3. Cloud-Based Management
The MRDP pattern "outsources" the heavy lifting of data management to external cloud services, specifically Globus, via REST APIs. This includes:
- Authentication: Using OAuth2 and OpenID Connect to allow users to log in with existing institutional identities.
- Transfer: Utilizing GridFTP for high-speed, reliable, third-party data movement that can be monitored and managed via the cloud.
- Sharing: Managing granular access permissions without requiring the creation of local system accounts for every user.
Real-World Performance and Scalability
The efficacy of the MRDP pattern is demonstrated through several high-impact deployments. For instance, the NCAR Research Data Archive (RDA) serves over 24,000 users and moves approximately 1.9 Petabytes of data annually. Transfers from NCAR to the NERSC supercomputing center have achieved sustained rates exceeding 1 GB/s (8 Gbps) over a 10 Gbps connection.
Other notable implementations include Argonne’s Petrel, which has recorded moving over 1 Petabyte of data in a single day, and the Sanger Imputation Service for genomic data. These examples show that by separating control logic from data storage, portals can remain responsive and secure even while massive datasets are transferred in the background.
Conclusion and Future Directions
The MRDP design pattern significantly reduces the technical debt of research teams while providing users with a high-performance, reliable experience. By leveraging cloud services and specialized network architectures, researchers can focus on science rather than infrastructure. Future work involves extending this pattern to include "on-demand" computation, where portals trigger containerized analysis (using Docker or Singularity) on data before it is delivered to the end user.
Summary updated