For decades, the standard workflow in data-driven science followed a simple, intuitive pattern: a centralized repository hosted a dataset, and researchers downloaded files to their local workstations to run analyses. This "data-shipping" model worked perfectly when datasets were measured in megabytes or gigabytes.
However, as scientific disciplines transition from the terascale to the petascale, this traditional model hits a physical wall. When a dataset grows to petabytes, it develops what is known as data gravity. The sheer mass of the information makes it highly immobile; the time, financial cost, and network bandwidth required to move it across the internet become prohibitive.
To sustain scientific discovery, the architecture must be inverted. Instead of moving the data to the compute, modern systems must move the compute to the data.
The Bottleneck of Data Gravity
Why does the traditional download model fail at scale? The core issue is a fundamental mismatch between the exponential growth of high-throughput data generation and the linear constraints of network bandwidth and local storage.
As Juliana Freire notes in a special issue editorial, this "data deluge" outpaces traditional data management tools, creating a critical bottleneck where the time spent transferring data can easily exceed the time required for actual scientific computation. For instance, moving a petabyte of data over standard networks can take weeks or months, effectively locking the data away from researchers who lack specialized high-performance connections.
This constraint introduces severe practical and equity issues across scientific domains:
- Resource Barriers: Processing massive datasets locally requires significant institutional investments in high-performance computing (HPC) hardware and local storage. As Florian Schintke et al. point out in their work on collaborative climate infrastructures, this dependency creates a steep barrier to entry for smaller research groups, limiting the scope of collaborative, multi-model studies.
- Data Silos: When transferring raw data is too slow, researchers are forced to work with isolated, localized subsets of data. As highlighted by Wei Ouyang et al. in the context of bioimage analysis, this leads to data silos where raw results cannot be easily explored or verified by the broader community until after processing is complete.
- Analysis Truncation: Faced with impossible transfer times, scientists often restrict their inquiries. Maria T. Patterson et al. observe in satellite imagery analysis that researchers frequently limit their scope to small geographic or temporal windows rather than analyzing entire historical archives.
Early attempts to mitigate this bottleneck focused on accelerating wide-area network transfers. For example, Robert L. Grossman et al. developed Sector, a decentralized peer-to-peer infrastructure that utilized dedicated optical lightpaths to achieve transfer speeds far exceeding the commodity internet. Yet, even with optimized network protocols, the sheer velocity of data generation in fields like genomics and remote sensing eventually outruns the capacity of physical networks.
The Mechanism: Inverting the Architecture
To overcome data gravity, the research community is shifting toward architectures that keep data stationary and bring processing power directly to the storage layer. This paradigm relies on three core pillars: federated server-side analysis, cloud-native data commons, and Evaluation-as-a-Service (EaaS).
Traditional Model (Data Shipping):
[Central Repository] ---- Petabytes of Data ----> [Local Workstation (Compute)]
*High Latency / Storage Bottleneck*
Compute-to-Data Model:
[Central Cloud / Data Commons]
├── [Stationary Petabyte Dataset]
└── [User Code / Containerized Algorithm] ----> Executed directly on storage nodes
Federated Server-Side Analysis
Instead of acting as passive file servers, modern data repositories function as distributed computational engines. In climate science, the Earth System Grid Center for Enabling Technologies (ESG-CET) transitioned from a standard repository to a globally federated, three-tier architecture. As described by D. N. Williams et al. in their data management framework and further detailed by Rachana Ananthakrishnan et al. in building the global federation, this architecture allows researchers to perform server-side analysis and filtering. Users request specific analytical operations or data reductions at the gateway level, and only the highly compressed results are transmitted back over the network.
Cloud-Native Data Commons
A data commons co-locates data, computing infrastructure, and core software tools into a unified, cloud-based ecosystem. In his synthesis of ten lessons for data sharing, Robert L. Grossman emphasizes that traditional repositories fail at the petabyte scale because they treat data as isolated files. A true data commons eliminates the financial and technical barriers of local hosting by allowing researchers to spin up virtual machines or cloud containers directly adjacent to the data.
To maximize efficiency within these environments, platforms must optimize how data is read. For example, the Matsu Wheel framework, detailed by Maria T. Patterson et al., addresses the input/output (I/O) bottleneck by batching multiple concurrent analytical queries together, scanning a massive dataset once and passing it through a pipeline of parallel computations.
Evaluation-as-a-Service (EaaS)
When datasets are too massive to move or too sensitive to share publicly (such as proprietary web crawls or confidential medical records), systems adopt an Evaluation-as-a-Service model. As summarized by Allan Hanbury et al. in an EaaS expert workshop report, this paradigm requires researchers to submit their algorithms (packaged as virtual machines or containers) to the infrastructure hosting the data. The platform executes the code locally, verifies the outputs, and returns only the performance metrics or high-level summaries to the researcher.
Real-World Implementations Across Domains
The transition to compute-to-data architectures is visible across a wide variety of data-intensive scientific fields, each adapting the concept to its unique data structures.
Seismology
Seismological data archives have grown rapidly due to continuous recording networks. Yiyu Ni et al. demonstrated the power of cloud-native workflows in a 2025 study by processing 1.3 petabytes of global seismic data directly on AWS. By moving their deep learning algorithms to the cloud storage bucket, they accomplished in three days an analysis that would have taken months over standard networks, picking 4.3 billion earthquake phases. In a companion guide for individual researchers, Yiyu Ni et al. emphasize that migrating these heavy workloads to commercial cloud instances bridges the gap between massive public archives and the limited desktop computing power available to solo scientists.
Geospatial and Remote Sensing
Geospatial platforms face unique challenges because satellite imagery is traditionally stored in fragmented, file-based formats. To bypass this, the IBM PAIRS Geoscope platform, designed by Conrad M Albrecht et al., indexes geospatial data at the pixel level and maps it into a specialized key-value store. As explained in their system design paper, this allows the platform to perform complex, multi-layer spatial analytics directly on the storage nodes, completely bypassing the need to download and align individual files.
Genomics and Bioimaging
High-throughput sequencing and advanced microscopy generate data at rates that outpace local storage capacity. Fábio C. P. Navarro et al. argue that genomics must be treated as a core data science subdomain, where workflows must inherently shift from local desktops to centralized cloud environments to remain sustainable.
To make these cloud environments accessible to biologists who may not be experts in cloud infrastructure, platforms are increasingly adopting web-based interfaces. The WebMeV platform, developed by Yaoyu E. Wang et al., provides a cloud-native environment where researchers can analyze large public datasets (like The Cancer Genome Atlas) using sophisticated tools through a browser interface, avoiding the latency and storage costs of local downloads.
The Future of Decentralized Science
By neutralizing data gravity, compute-to-data architectures do more than just accelerate computation; they democratize access to big data. When petabyte-scale datasets are paired with cloud-adjacent compute, a researcher at a small institution has the same analytical capability as one at a major research university.
As communities combine these cloud-native platforms with standardized data structures and containerization—a trend highlighted by Russell A. Poldrack et al. in neuroimaging reproducibility—scientific workflows become fully transportable, automated, and verifiable. The ultimate goal of this shift, as articulated in German climate research initiatives like C3Grid, is a seamless, distributed environment where data management and scientific computation merge into a single, unified pipeline.
Key Foundational Work
Data management and analysis for the Earth System Grid (2008) — contributed 1 concept elements to this direction.
Building a global federation system for climate change research: the earth system grid center for enabling technologies (ESG-CET) (2007) — contributed 1 concept elements to this direction.
A peer-to-peer infrastructure for distributing large scientific data sets over wide area high-performance networks: experimental studies using wide area layer 2 services (2007) — contributed 1 concept elements to this direction.