Distributed storage systems rely on erasure codes to protect data against hardware failures without incurring the massive storage overhead of simple replication. Among these, Locally Repairable Codes (LRCs) have become an industry standard because they minimize the cost of repairing a single failed node. They do this by dividing data into small, local groups; if one node fails, it can be reconstructed by downloading data from just a few neighboring nodes within its local group, rather than reading the entire dataset.
However, real-world data centers frequently experience concurrent failures due to rack outages, power spikes, or batched maintenance operations. As Yanbo Lu et al. point out in On the Single-Parity Locally Repairable Codes with Multiple Repairable Groups, traditional LRC frameworks are fundamentally optimized for isolated, single-node failures. When multiple nodes within the same local group fail simultaneously, the local parity equations become underdetermined. The system is forced to fall back on costly global decoding, which floods the network with traffic and triggers massive disk I/O across the entire cluster (Yanghuang Cao et al.).
To bridge this gap, a prominent research direction focuses on extending local repairability to handle multiple concurrent failures efficiently. This is achieved through two primary paradigms: Cooperative Local Repair and Centralized Multi-Node Repair.
Cooperative Local Repair: Sharing the Recovery Burden
The core intuition behind cooperative local repair is that failed nodes should not act in isolation. If multiple nodes are down, they can collaborate during the recovery process to eliminate redundant data transfers.
Ankit Singh Rawat et al. formalized this framework by introducing $(r, \ell)$-cooperative locality in On cooperative local repair in distributed storage. and Cooperative local repair in distributed storage. In this model, any subset of $\ell$ failed nodes can be jointly reconstructed by accessing a small, shared support set of at most $r$ healthy helper nodes.
Cooperative repair achieves significant bandwidth savings through two main coordination mechanisms:
- Shared Helper Downloads: Instead of each failed node downloading independent datasets from the helpers, the helpers generate combined data packages. In A New Cooperative Repair Scheme with k + 1 Helper Nodes for (n, k) Hadamard MSR codes with Small Sub-packetization, Yajuan Liu et al. demonstrate a scheme where helpers transmit shared sums of paired symbols. The data retrieved to fix one node simultaneously contains the algebraic components needed to reconstruct another, keeping total network traffic at the theoretical minimum.
- Lateral Communication (Two-Phase Recovery): The authors of A Class of Minimum Storage Cooperative Regenerating Codes with Low Access Property propose a two-phase strategy. First, the failed nodes download a minimal set of data from surviving helpers. Second, they exchange processed intermediate information directly among themselves. This lateral exchange bypasses the core storage network, resolving the local multi-failure bottleneck without overloading the surviving infrastructure.
Centralized Multi-Node Repair: Joint Linear Optimization
When failed nodes cannot communicate directly with each other, coordination must happen at a single location—such as a master storage controller or a dedicated replacement rack. This is known as Centralized Multi-node Repair (CMR), a framework established by Ankit Singh Rawat et al. in Centralized Repair of Multiple Node Failures With Applications to Communication Efficient Secret Sharing.
Independent Repair (Inefficient):
Helper Nodes ------> Failed Node 1 (Downloads subset X)
Helper Nodes ------> Failed Node 2 (Downloads overlapping subset X)
Centralized Repair (Efficient Joint System):
Helper Nodes ------> Central Controller ------> Reconstructed Node 1
(Downloads X once, ------> Reconstructed Node 2
solves joint matrix)
In a centralized setting, bandwidth savings come from solving the recovery of multiple nodes as a single, joint linear system rather than running independent single-node repair routines. In Repairing Multiple Failures for Scalar MDS Codes, Burak Bartan et al. extend the "repair matrix" concept to scalar Maximum-Distance Separable (MDS) codes. Because the central controller manages the entire recovery operation, it can identify overlapping information required by different failed nodes. By downloading a single, carefully chosen linear combination of data from a surviving helper, the controller satisfies the requirements of multiple recovery equations at once, drastically reducing the total repair bandwidth.
Structural Innovations in Code Design
Enabling local recovery for multiple failures requires fundamental changes to how erasure codes are structured. Traditional LRCs use a single local parity check per group, which provides a local distance of 2 (allowing only one local failure to be corrected).
To handle multiple failures, researchers have introduced two primary structural innovations:
- Higher Local Distance and Multi-Locality: Codes must be designed with more robust local properties, such as the multiple $(r_i, \delta_i)$-localities explored by Bin Chen et al.. Increasing the local distance allows a group to tolerate multiple internal failures. While analyzing these advanced structures introduces significant mathematical complexity (Jie Hao et al.), it allows systems to maintain local repairability even during dynamic operations like storage rate conversion (Songping Ge et al.).
- Dual Group Participation: In TFR-LRC: Rack-Optimized Locally Repairable Codes: Balancing Fault Tolerance, Repair Degree, and Topology Awareness in Distributed Storage Systems, Yanghuang Cao et al. propose a topology where every data and global parity block participates in two distinct local parity groups. This overlapping design creates redundant repair paths. If a multi-node failure overwhelms one local group, the system can route the recovery of the surviving blocks through their second local group, avoiding a costly global decoding operation.
Application Beyond Hard Drive Arrays
The principles of multi-failure local repair are highly adaptable and extend to other distributed environments:
- Wireless Storage Networks: In wireless settings, point-to-point communication is costly. Nitish Mital et al. demonstrate in Functional Broadcast Repair of Multiple Partial Failures in Wireless Distributed Storage Systems that the physical broadcast nature of wireless signals can be exploited. A single transmission from a surviving helper node can be received by multiple failed nodes simultaneously, naturally optimizing the cooperative repair process.
- Network Function Virtualization (NFV): Beyond data storage, Jörg Kliewer applies these coding principles to virtualized network functions running on commodity hardware in Coded Network Function Virtualization: Fault Tolerance via In-Network Coding, showing that in-network coding significantly increases the number of concurrent server failures a system can tolerate before experiencing downtime.
- Retrofitting Existing Systems: Implementing these techniques does not always require a complete system overhaul. Runhui Li et al. demonstrate in CORE: Augmenting Regenerating-Coding-Based Recovery for Single and Concurrent Failures in Distributed Storage Systems that a "virtual symbol" reconstruction layer can be overlaid on top of existing single-failure codes. This layer intercepts multi-node failure events and coordinates their joint recovery, delivering up to a 64% reduction in repair traffic without altering the underlying data layout.
Key Foundational Work
On cooperative local repair in distributed storage. (2014) — contributed 1 concept elements to this direction.
CORE: Augmenting Regenerating-Coding-Based Recovery for Single and Concurrent Failures in Distributed Storage Systems (2013) — contributed 1 concept elements to this direction.