Letting Each Federated Client Decide How Much to Trust a Sentiment Lexicon

A sentiment lexicon is a hand-built dictionary that marks words as positive or negative. It is the one piece of sentiment knowledge that federated learning (training a shared model across many data holders who never pool their raw text) gets for free: it is public, so every client can hold a copy without any privacy cost. Lexicons help most exactly where federated clients are weakest, on small, sparse local vocabularies. They also hurt in a predictable way. A word's polarity depends on the domain ("unpredictable" is praise for a thriller and a complaint about brakes), so a lexicon that sharpens one client's embeddings can quietly corrupt another's. Methods that add lexicons to word embeddings set a single, hand-tuned weight on the lexicon. In a federation, where each client's domain differs, one global weight is the wrong object to tune, and tuning a separate weight per client with ordinary search costs many full federated training runs.

We propose that each client learns its own lexicon weight during training, at almost no cost, by asking one question at every local step: does pulling my embeddings toward the lexicon's polarities make my held-out data easier to classify right now? The answer is a single dot product between two gradients. The method removes the outer tuning loop for this hyperparameter. It also gives a measurable diagnostic of where the lexicon is miscalibrated for a domain, and it should show clients drifting toward the lexicon early and away from it once local data speaks for itself.

Core Claim

The lexicon-strength hyperparameter should be local to each client and should change over training, and a one-step hypergradient is enough to learn it. A hypergradient is the derivative of validation loss with respect to a hyperparameter. Concretely, client $i$ minimizes $L_i(\theta) + \lambda_i R(\theta)$, where $\theta$ are the shared word embeddings and classifier, $L_i$ is the local classification loss, and $R$ is a lexicon regularizer. We use the logistic SentiVec loss from Searching for the X-Factor, which pushes words of known opposite polarity to opposite sides of a learned hyperplane. After a local step $\theta_{t+1} = \theta_t - \eta(\nabla L_i(\theta_t) + \lambda_i \nabla R(\theta_t))$ with step size $\eta$, differentiating a small local validation loss $V_i$ through that one step gives

$$\frac{\partial V_i(\theta_{t+1})}{\partial \lambda_i} = -\eta\, \nabla V_i(\theta_{t+1})^\top \nabla R(\theta_t).$$

Client $i$ therefore raises $\lambda_i$ when the lexicon's pull points the same way as the validation gradient's descent direction, and lowers it when they conflict. The update is $\lambda_i \leftarrow \max(0, \lambda_i - \beta \cdot \text{normalized hypergradient})$, where $\beta$ is a meta step size. This is the same inner-product trick that Online Learning Rate Adaptation with Hypergradient Descent uses for the learning rate, with the gradient of the lexicon regularizer in place of the previous training gradient. Its cost is one extra backward pass on a small validation batch. Because $\nabla R$ is nonzero only on lexicon rows, the dot product is cheap and sparse.

The claim has three testable parts. First, per-client adaptive $\lambda_i$ matches or beats the best single global $\lambda$ found by expensive federated search. Second, it does so in one training run. Third, and most distinctive, $\lambda_i$ falls specifically on clients whose domain contradicts the lexicon. A simple heuristic based on lexicon coverage or client data size cannot produce that behavior.

Relation to Prior Work

Three centralized papers motivate the problem. SINAI at IEST 2018 concatenated emotion-lexicon indicators with Twitter GloVe vectors in LSTMs. Lexicon features lowered development-set scores but improved the unseen test set, which is evidence that fixed-strength lexical injection trades early fit for later generalization. A schedule that changes over time might recover both. BCSAT appended hand-set polarity counts to domain Word2Vec vectors for Telugu and raised SVM accuracy from 84.08% to 86.57%, but the fusion was fixed by hand. SentiVec made the lexicon part of the embedding objective, and a lexicon-regularized Wikipedia model rose from 81.6% to 82.4% average accuracy on Amazon reviews. The same paper showed that corpus subjectivity, meaning domain, strongly shapes what embeddings learn, and its lexicon weight is still manual. We reuse SentiVec's regularizer as the mechanism. What changes is who sets its weight and when.

On the tuning side, several methods constrain and inform the design. FedHyper applies hypergradient descent to global and local learning rates in federated learning with under 5% overhead and reports 15% higher Shakespeare accuracy than FedAvg from poor initial rates. It tunes only step sizes, never a regularizer that carries external knowledge. AggITD computes full bilevel hypergradients in a communication-efficient way, but it is sensitive to Hessian conditioning and has been tested only on small vision models. We deliberately use the cheaper one-step approximation and treat AggITD as a heavier alternative, not a component. FedPop tunes a population of configurations during training, at the cost of storing several models on the server. HPN personalizes hyperparameters with a policy network conditioned on random-feature sketches of client data. Those sketches capture input statistics, not whether the lexicon agrees with the client's domain. FedHPO-Bench gives the evaluation harness and finds Gaussian-process Bayesian optimization a consistently strong black-box tuner.

Finally, Federated Learning of Medical Concepts Embedding using BEHRT shows that federated domain embeddings approach centralized quality. Its gap to centralized training was largest, 3% absolute, on the sparsest cohorts, and it used no external knowledge and relied on grid search. That regime is where we expect lexical anchors with adaptive weights to matter most.

Research Program

We will build a federated sentiment classifier whose embedding layer and small encoder (a BiLSTM, chosen so that embedding geometry drives accuracy) are shared and averaged with FedAvg. Each client keeps a private scalar $\lambda_i$ and a private validation slice of about 10% of its data. The lexicon is SentiVec's polarity list, held by every client.

The first study characterizes the schedules the method learns. It asks whether $\lambda_i$ rises early and decays late, the "mountain" shape FedHyper observed for learning rates, and whether this recovers SINAI's test-set gain without its development-set loss. The second study tests domain sensitivity. On clients defined by product category, we will correlate the learned $\lambda_i$ with an independent measure of lexicon–domain disagreement: the fraction of lexicon words whose sign flips in a logistic regression fit on that client's labels. The third study extends to a grouped variant, $\lambda_{i,k}$, with one weight per cluster of lexicon words. This asks whether the method can switch off only the polysemous words while keeping the rest. Throughout, the meta step size $\beta$ is a new hyperparameter. We will report sensitivity across two orders of magnitude and use the normalized update to keep it scale-free, as the hypergradient-descent paper recommends.

Baselines

The naive reference is FedAvg with no lexicon at all, trained with the same model, rounds, and learning-rate search. If this is within noise of every lexicon variant, the question is moot on that dataset and we will say so.

The strongest global competitor is FedAvg with SentiVec at a single global $\lambda$ tuned by Gaussian-process Bayesian optimization inside FedHPO-Bench, the tuner that paper found most reliable. For fairness it gets the same total client compute as our method multiplied by the number of trials it uses. We also report it at matched compute, which allows only one or two trials, to show the overhead trade-off directly. We beat it if we match its best accuracy in a single run, or exceed it on worst-client accuracy.

The strongest personalized competitor is HPN with $\lambda$ added to its search space alongside learning rate, run with its random-start training protocol. HPN reports 60.32% accuracy on the federated Twitter task, which is the external number our Twitter runs must reproduce under its setup before comparison. We beat it if we exceed that setup's accuracy at lower tuning compute. FedPop with $\lambda$ in its population serves as the in-training evolutionary alternative and is charged for its server memory. For Amazon-by-category federated partitions, no reported result was found on Lacuna, so all numbers there come from our own reruns. The centralized SentiVec figures (81.6% to 82.4%) serve only as a sanity reference for the centralized upper bound.

Two ablations isolate the key ingredient. The first applies FedHyper's learning-rate hypergradients with a fixed, tuned $\lambda$. If it matches our method, the gain came from step-size adaptation and not from lexicon weighting. The second trains with a lexicon whose polarities are randomly shuffled. A correct method must drive $\lambda_i$ to near zero and lose nothing relative to the no-lexicon baseline.

Why not just…?

Why not just set each client's lexicon weight from lexicon coverage or data size? This is the most serious threat. Small clients plausibly need more prior, so $\lambda_i \propto$ coverage$/\sqrt{n_i}$ may capture much of the benefit at zero cost. We include this heuristic as a baseline, fitting its single scale constant with the same Bayesian-optimization budget. The experiment that separates the two is planted miscalibration: on a subset of clients we flip the labels' relationship to a chosen set of lexicon words, leaving coverage and data size unchanged. The heuristic cannot react. The hypergradient should lower $\lambda$ on exactly those clients. If the heuristic matches us on natural data and the gap appears only under planted corruption, we will report the method as a safeguard, not an accuracy gain.

Why not just tune one global weight with standard federated search? FedHPO-Bench makes this easy, and if domains agree with the lexicon, one value may suffice. The Amazon-by-category partition tests whether they agree. SentiVec's own finding that domain subjectivity changes embedding quality suggests they do not. A small spread in learned $\lambda_i$, together with no worst-client gain, would count against our claim.

Why not just skip lexicons and fine-tune a pretrained transformer? Pretrained encoders may absorb most of what a lexicon offers. Pàmies et al. found lexicon features still sharpen BERT representations, but that was centralized. We run a DistilBERT-scale arm with and without the lexicon regularizer on its input embeddings, and we report its per-round communication cost. If the lexicon gives nothing there, we will scope the claim to compact on-device encoders, where transmitting a transformer every round is impractical.

Proposed Experiments

The main benchmark partitions the 24-category Amazon review corpus from SentiVec into 24 clients, one per category, with a low-data regime that subsamples each client to between 200 and 2,000 reviews. The secondary benchmarks are FedHPO-Bench's Twitter and SST-2 federated tasks, which allow direct comparison with HPN and the benchmark's tuners. A stress test adds one very small client built from BCSAT's 668 Telugu reviews with its lexicon, where the prior should matter most. Metrics are mean and worst-client accuracy, total client gradient evaluations including all tuning trials, communication rounds to reach a target accuracy, and the correlation between learned $\lambda_i$ and measured lexicon–domain disagreement.

Three results would be decisive. The first is the planted-corruption test above, where success means $\lambda$ drops selectively on the corrupted clients while coverage-based weights do not. The second is the shuffled-lexicon test, where success means $\lambda$ falls toward zero with no accuracy loss. The third is matched-compute comparison against Bayesian-optimization-tuned global $\lambda$ and HPN, where success means equal or better accuracy from a single run. If the first two pass but the third shows no gain, the contribution is a cheap, interpretable per-domain audit of a lexicon, not a better classifier, and we will report it in those terms.