Abstract
In this paper, we analyze a highly distributed backup storage system realized by means of nano datacenters (NaDa). NaDa have been recently proposed as a way to mitigate the growing energy, bandwidth and device costs of traditional data centers, following the popularity of cloud computing. These service provider-controlled peer-to-peer systems take advantage of resources already committed to always-on set top boxes, the fact they do not generate heat dissipation costs and their proximity to users. In this kind of systems redundancy is introduced to preserve the data in case of peer failures or departures. To ensure long-term fault tolerance, the storage system must have a self-repair service that continuously reconstructs the fragments of redundancy that are lost. In the literature, the reconstruction times are modeled as independent. In practice, however, numerous reconstructions start at the same time (when the system detects that a peer has failed). We propose a new analytical framework that takes into account this correlation when estimating the repair time and the probability of data loss. We show that the load is unbalanced among peers (young peers inherently store less data than the old ones). The models and schemes proposed are validated by mathematical analysis, extensive set of simulations, and experimentation using the GRID5000 test-bed platform. This new model allows system designers to operate a more accurate choice of system parameters in function of their targeted data durability.
Introduction
In distributed storage systems, such as peer-to-peer backups or "nano-datacenters," data is protected by redundancy—either through simple replication or erasure coding. When a node fails, the system must "self-repair" by reconstructing the lost data fragments on new nodes.
A major challenge in these systems is accurately calculating the time required for these repairs. Most existing models assume repairs are independent events following an exponential distribution. In reality, when a disk fails, it triggers a massive burst of reconstruction tasks simultaneously. These tasks compete for the same limited upload bandwidth of the surviving peers. Furthermore, "young" nodes that recently joined the system have less data than "old" nodes, creating a workload imbalance. Ignoring these factors leads to "naive" calculations that can underestimate repair times by orders of magnitude, resulting in a false sense of data security.
Methodology
The authors propose a discrete-time Markovian queuing model to track the "upload load" of the global system. The state of the system is defined by the number of fragments currently awaiting reconstruction.
Key components of the model include:
- Arrival Process: Node failures are modeled as Poissonian batch arrivals. When a node fails, all fragments it stored enter the queue at once.
- Service Rate: The system's ability to process these repairs is limited by the aggregate upload bandwidth ($BW_{up}$) and a "bandwidth efficiency factor" ($\rho$).
- Disk Asymmetry Modeling: A truncated geometric distribution is used to model disk occupancy, accounting for the mix of nearly empty new disks and nearly full old disks.
- Regenerating Codes: The model specifically evaluates Minimum Bandwidth Regenerating (MBR) codes, designed to minimize data transfer during repair.
Validation and Results
The researchers validated their theoretical model using mathematical analysis, custom cycle-based simulations, and real-world experimentation on the Grid’5000 platform using 64 nodes. They utilized failure traces from PlanetLab and Microsoft PCs to set realistic Mean Time To Failure (MTTF) parameters.
Key Insights
- The Naive Gap: A "back-of-the-envelope" calculation might suggest a repair takes 17 hours, while the proposed model (and real-world tests) shows it actually takes 9 days. This difference changes the Probability of Data Loss Per Year (PDLPY) from a negligible $10^{-8}$ to a catastrophic $0.2$.
- Load Imbalance as a Bottleneck: Because old disks are fuller, they are more likely to be involved in repairs. If a repair requires data from a specific "full" disk, that disk becomes a bottleneck for the entire reconstruction process.
- The "Degree of Freedom" Paradox: Counter-intuitively, using a less bandwidth-efficient code can sometimes result in faster repair times. By reducing the "repair degree" ($d$), the system gains more freedom to choose which peers participate in a repair, bypassing heavily loaded nodes.
Conclusion
The analytical model's predictions for average repair time closely matched simulation and experimental results. In real-world 64-node tests, the experimental mean repair time (148 seconds) was nearly identical to the simulator's prediction (145 seconds), proving the model holds up under actual network congestion. This framework allows system designers to make more accurate choices regarding system parameters to meet targeted data durability goals.
Summary updated