Sovereignty as Information Flow in Decentralized Fine-Tuning

Indigenous communities that want language technology usually have to choose between two options. One is to hand their recordings and texts to a central lab, which trains a strong model but takes physical control of the data. The other is to train alone on a few thousand sentences, which keeps control but gives up whatever benefit related communities' data could offer. Decentralized learning, where devices train locally and exchange model updates instead of raw data, looks like a way out. It has a hidden flaw, though. Community data-sharing rules are relational: "our stories may help our sister village's model, but not anyone outside the region." Gossip-style decentralized training passes information along multiple hops. If village A shares with B and B shares with C, A's knowledge reaches C even though A never allowed it. Removing the forbidden link from the network does not prevent this.

We propose to treat community governance rules as information-flow labels, borrowing the idea from computer-security research on tracking where data may travel. Every piece of model state carries the set of communities allowed to see it. Each model component, a small low-rank adapter added on top of a frozen translation model, may be trained only from sources whose permissions include its whole label, and may be sent only to communities inside that label. Under this protocol, permission leaks across multiple hops cannot occur, by construction. The price is that the number of possible labels grows exponentially with the number of communities. Choosing which few labels to actually instantiate then becomes a budgeted selection problem. We solve it with the kind of polynomial-time combinatorial search used in edge federated learning. The claim is that this gives an exact sovereignty guarantee at the level of model parameters. It should also recover a measurable share of the translation-quality gain from sharing, at a communication and memory cost that grows with the number of selected labels rather than the number of communities.

Why this gap is real

The critical literature has established what the protocol must respect. Without consent, scraping Indigenous corpora for large models repeats colonial extraction (Data-mining and Extraction: the gold rush of AI on Indigenous Languages; The Ethical Question – Use of Indigenous Corpora for Large Language Models). Calling such languages "low-resource" hides the political causes of scarcity and the fact that the data belongs to sovereign peoples (Primum Non Nocere; Decolonising Speech and Language Technology). The most concrete constraints come from the interview study in Ethical Considerations for Machine Translation of Indigenous Languages. It surveyed 22 activists from more than 13 groups and found that nearly half want ownership to rest with speakers or the community, that only 17% are comfortable with research groups holding data, that 45.5% want poor or experimental systems kept private, and that participants fear central systems will erase dialect differences. These papers motivate the work. None of them proposes a training procedure.

Infrastructure work supplies the label source but stops at the archive boundary. The Language Data Commons of Australia (Access Control Framework for Language Collections) attaches machine-readable licenses and Traditional Knowledge labels to files. It enforces them through identity checks at download time and has no say over what a model trained on the data can later reveal. The Esethu dual-licensing scheme (The Esethu Framework) governs commercial use contractually, then fine-tunes a corporate speech model centrally. The LiFE Suite (Field to Model) gives communities ownership at the collection step. Its own authors note, however, that annotation and training depend on cloud services and stable internet. We take these metadata labels as the input to our protocol. Our contribution is to carry them through training.

The optimization literature supplies the methods, but its assumptions are wrong for this setting. Over-the-air decentralized learning (Decentralized Federated Learning via MIMO Over-the-Air Computation) relies on a symmetric, doubly stochastic mixing matrix, a weighting of neighbors that conserves total influence in both directions. Its physical superposition of signals mixes every transmitter together, so it can only be used inside a single permission group. Asymmetrically Decentralized Federated Learning handles directed links with Push-Sum, a gossip protocol that corrects for uneven numbers of neighbors. It adds sharpness-aware minimization for heterogeneous clients and proves an $O(1/\sqrt{T})$ rate. Its goal, though, is consensus, where every node converges toward the same model. Consensus over a connected graph is exactly multi-hop leakage. Joint Edge Aggregation and Association (EARTH) supplies the combinatorial technique we reuse. It decomposes a mixed-integer problem and runs a two-stage local search under a matroid constraint, staying within 0.51% of brute force. Its objective, however, counts only time and energy.

Core Claim

Let $V$ be the set of communities. Each community $i$ holds data $D_i$ with a permission set $L_i \subseteq V$, the communities allowed to benefit from it, compiled from its metadata and licenses. A model component is an adapter $A_P$ tagged with label $P \subseteq V$.

The protocol enforces one rule. $A_P$ may reside only at nodes in $P$. Its updates may read only from data $D_i$ with $P \subseteq L_i$ and from adapters $A_Q$ with $Q \supseteq P$. When community $j$ translates, it uses the frozen base model plus every adapter whose label contains $j$. Combining information therefore never widens who may see it, and every parameter at node $j$ is a function only of data whose owners permitted $j$.

There is a subtlety that a naive "per-source adapter" scheme misses. The rule constrains the forward pass used to train $A_P$, not only where $A_P$ is sent. If $A_P$ were trained with an adapter carrying a narrower label active, knowledge from that narrower group would flow into $A_P$ through the gradients and leak to every member of $P$.

The scalability question is which labels to instantiate. A label $P$ lets data from community $i$ reach community $j$ when $P \subseteq L_i$ and $j \in P$. As a first-order proxy for usefulness, we pick a family $\mathcal{S}$ of labels to maximize permitted data coverage under a communication budget:

$$\max_{\mathcal{S}} \sum_{(i,j)} |D_i|\, \mathbf{1}\big[\exists P \in \mathcal{S}: P \subseteq L_i,\ j \in P\big] \quad \text{s.t.} \quad \sum_{P \in \mathcal{S}} c\,|P| \le B,$$

where $c$ is the per-round cost of sending one adapter to one node and $B$ is the round budget. This is weighted budgeted coverage, so the objective is submodular, meaning additional labels help less as more are chosen. Greedy selection therefore carries the standard constant-factor guarantee, and EARTH-style swap search can refine it. Coverage is only a proxy. One adapter pooling many unrelated languages may help less than coverage suggests. We will therefore measure how well coverage predicts translation quality rather than assume it does.

Research Program

The first component is the protocol itself. It is built on LoRA fine-tuning of NLLB, a large multilingual translation model, following BSC Submission to the AmericasNLP 2024 Shared Task. That system updated 14.2% of parameters and won Spanish to Quechua and Spanish to Guarani.

Each label-$P$ adapter is trained by decentralized averaging among nodes in $P$ that hold eligible data. Averaging happens only inside $P$, so a symmetric mixing matrix restricted to $P$ is legitimate there. Over-the-air aggregation also becomes a permitted option within a label group. Where intra-group links are directed, we use Push-Sum with sharpness-aware updates as in DFedSGPSM. Training schedules respect the reading rule: an adapter is trained only with adapters of broader-or-equal labels active.

The second component is label selection: greedy and local search over the lattice of candidate labels, with cost measured in bytes per round and device memory.

The third component is a policy generator. Without real community partners we cannot use actual policies, so we generate families grounded in the categories Mager et al. report. These include fully open policies, item-level restrictions (a fraction of sentences marked sacred or ceremonial and allowed nowhere), regional clusters, and asymmetric policies (A permits B, B does not permit A). The fraction of relational versus item-level constraints is varied explicitly, because that ratio decides whether the method is needed at all.

A final phase, outside the scope of the benchmark claims, would replace synthetic policies with ones authored by community partners through LDaCA-style metadata.

The guarantee has stated limits. It assumes nodes run the protocol honestly. It is a dataflow guarantee, not cryptography, and it says nothing about an authorized community re-sharing its own model. It also leaves the dependence on Meta's base checkpoint, which the file flags as an open tension, untouched.

Baselines

The naive reference point is the frozen NLLB model with no adaptation. No reported zero-shot NLLB result for the AmericasNLP 2024 test set was found on Lacuna, so we will measure it ourselves. The second reference is local-only LoRA, where each community trains its own adapter and shares nothing. This is the strongest sovereignty-respecting competitor, because it trivially satisfies every policy at zero communication cost. Beating it is the whole case for sharing.

The unconstrained upper reference is centralized LoRA on pooled data, run with the BSC recipe. BSC reports 38.21 ChrF++ for Spanish to Quechua and 38.93 ChrF++ for Spanish to Guarani (ChrF++ is a character-level overlap score). The NordicsAlps system ranked first on average across languages (System Description of the NordicsAlps Submission). Its subword tokenization is orthogonal to our method and could be applied to every arm. We report the gap to pooling as the price of sovereignty, not as a target to beat.

The strongest decentralized competitor is DFedSGPSM run on the communication graph with all forbidden edges deleted. Plain decentralized averaging on the same pruned graph is also included, for the case where links are symmetric. The fourth baseline filters out item-restricted data and then runs standard decentralized training freely.

Every arm uses the same base checkpoint, LoRA rank, total local training steps, and hyperparameter search budget, tuned on development sets only. Our method's adapter budget is matched to a fixed total of communicated bytes.

The key ablation removes the reading rule. Labeled adapters are routed correctly, but each is trained with all locally present adapters active. This isolates whether the gradient-path constraint matters. A second ablation compares greedy label selection with instantiating every feasible label, which is exact but expensive.

Beating the competitors means two things. Against local-only, we need higher mean ChrF++ across communities with zero measured leakage. Against pruned-graph gossip, we need equal-or-better ChrF++ at forbidden-pair leakage indistinguishable from chance.

Why not just…?

Why not just rely on licenses and Traditional Knowledge labels? These mechanisms govern who downloads a file. They do not govern what a trained model reveals afterward, and Access Control Framework for Language Collections relies on identity checks that end at access. Junker documents developers ignoring non-binding declarations. We do not compete with metadata. We compile it into labels, so the question is only whether training preserves it, and the leakage experiment answers that.

Why not just pool data centrally with LoRA? It will probably translate best, and we expect it to stay ahead. However, it requires communities to surrender possession of their data, which a large share of Mager et al.'s respondents reject. Our experiments report exactly how many ChrF++ points pooling buys, so communities can weigh that cost themselves.

Why not just train separate local adapters? This is the real threat. The eleven AmericasNLP languages come mostly from different families, and cross-language transfer may be small, apart from what the shared Spanish source side contributes. If labeled sharing does not beat local-only by more than run-to-run variance under realistic policies, we will report that sharing is not worth its complexity for this setting. In that case the contribution would shrink to the leakage finding about gossip.

Why not just filter restricted items and then train freely? If all policies are item-level ("never use sacred texts"), filtering is exact and simpler, and our method reduces to it. Our method is needed only when policies are relational. The policy-mix sweep therefore reports the crossover point, the share of relational constraints at which filtering begins to leak or must discard too much data.

Why not just delete forbidden edges and run Push-Sum? Multi-hop mixing still carries A's information to C through B. The canary experiment is designed to show this directly. If measured exposure turns out to be negligible in practice, for example because averaging washes canaries out, that would weaken our motivation, and we would report it.

Proposed Experiments

The testbed uses the AmericasNLP 2024 data (Findings of the AmericasNLP 2024 Shared Task), with 1,003 test sentences per language (1,000 for Chatino) and ChrF++, ChrF, and BLEU as metrics. The eleven languages serve as nodes. A second configuration splits languages with multiple training sources into several simulated communities, so that we can study regional-cluster policies among closely related data.

For leakage, we insert unique canary sentence pairs into each community's training data. We then measure canary exposure, meaning how much more likely a model makes the inserted sentence than comparable sentences it never saw, along with membership-inference accuracy, at every node the owner did not permit.

For cost, we record bytes per round, number of adapters held per device, and wall-clock time. These are measured on edge-class hardware with the smaller distilled NLLB variant, to test the connectivity concern raised by LiFE.

The decisive checks are three. First, pruned-graph Push-Sum shows above-chance canary exposure at forbidden nodes, while our protocol shows chance-level exposure, and so does the full protocol but not the ablation without the reading rule. Second, under policies with a substantial relational share, our protocol beats local-only adapters in mean ChrF++ while staying within the byte budget. Third, ChrF++ as a function of the number of selected labels $K$ saturates at small $K$, which shows that label selection, not the exponential label lattice, determines cost.

A negative outcome on the second check with a positive one on the first would still be informative. It would show that standard decentralized learning is unsafe under relational sovereignty rules, and that for these languages the safe alternative is simply to train locally.