Many practical agents are judged on several hand-designed reward terms at once, such as speed, energy use, and stability. Nobody knows how much each term should count. A common default comes from the decision-making literature: let the data set the weights, and give more weight to terms that vary more across the candidates being compared, on the theory that a term everyone scores the same on tells you nothing. This note argues that the default breaks exactly when it is most needed, in deployments where some measurement channels suffer bursty, drifting noise. A channel that is loud rather than informative looks maximally "discriminating," so it collects the most weight, and the ranking of candidate policies ends up following the noise.
We propose a weighting rule that splits each term's variation into two parts. The first is spread between candidates, which is real disagreement about which policy is better. The second is spread within a candidate over repeated evaluations, which is hesitancy about how good that policy is. The second part is tracked over time, the way noise-robust speech recognizers track the signal-to-noise ratio frame by frame. Each candidate's per-term evaluation is stored as a probabilistic hesitant fuzzy element, meaning a small set of possible scores, each with a probability, instead of a single number. That representation keeps the within-candidate spread that ordinary averaging throws away. The claim to test is concrete. Under non-stationary noise, this rule keeps its policy rankings and weights close to what it would compute from clean evaluations. Entropy weighting, running-variance normalization and smoothed rewards do not.
Core Claim
The setting is choosing, and optionally training, a policy when reward-term weights are unknown. It is framed as multi-attribute decision making: the alternatives are candidate policies, such as a Pareto set from a multi-objective learner or a sweep of checkpoints, and the attributes are the reward terms. Three pieces of prior work motivate the claim. First, Asudeh et al. show that small weight changes reorder rankings entirely, so the weighting step carries real consequences. Second, entropy weighting as used by Wang et al. derives objective weights from how scores are spread across candidates. Zhu applies the same idea directly to a probabilistic dual hesitant matrix. Third, both assume a static, noise-free matrix, so spread caused by corruption counts as information.
The rule we propose is an estimator, not a new theory of preference. For term $j$, let $B_j$ be the variance across candidates of their credibility-weighted mean scores. Let $N_j(t)$ be the within-candidate variance of term $j$ estimated in the time window around $t$, updated with a forgetting factor. The weight is
$$w_j(t) \propto \frac{B_j(t)}{B_j(t) + N_j(t)} \cdot \frac{B_j(t)}{\sum_k B_k(t)},$$
normalized to sum to one. The first factor is the fraction of the term's variation that is between-candidate signal. The second keeps the entropy-style preference for terms that separate candidates.
Two further ingredients carry the non-stationary part. The first is a blocked evaluation schedule. All candidates are evaluated in interleaved episodes within the same short time block, so noise that is shared across candidates within a block can be estimated from the common component and removed from differences between candidates. This is the reinforcement learning analogue of the instantaneous frame-level SNR in Dong Yu et al.'s variable-parameter HMMs, which lets the acoustic model adapt to momentary noise conditions instead of utterance-level averages. The second is per-episode credibility. Episodes collected in high-noise blocks enter the hesitant element with lower probability mass. This borrows the credibility idea from Guo, which down-weights hesitant experts in a static panel. Here it is recomputed as noise conditions change.
How This Differs From Closest Work
The nearest work is Zhu's entropy-weighted, prospect-theory ranking over probabilistic dual hesitant elements. That paper derives unknown weights from a hesitant matrix, but it uses one static matrix of three simulated experts, and its entropy weights reward spread of any origin. Guo adds credibility and power averaging against outlier experts, again on a static synthetic matrix, and acknowledges that the field relies on synthetic validation. Lian et al. blend objective and interval-bounded subjective weights through relative entropy, which has the same static assumption. We keep their representation and operators and change what the weight measures: between-candidate signal relative to time-local noise. We also test the rule on streams of reinforcement learning evaluations with real noise dynamics rather than on hand-written matrices.
On the reinforcement learning side, Inverse Reward Design infers a posterior over the designer's intended weights. That is a question about intent. We make no claim to recover intent. Our goal is narrower: the weighting default that engineers already use should not be hijacked by noise. Roijers's convex coverage set planning, PSL-MORL and C-MORL avoid choosing weights during training by learning a whole front. They supply our candidate sets, but they still leave a selection step, and they assume clean, stationary reward vectors. Adaptive Reward Design adapts shaping when completion rates drop. It cannot tell a drop caused by transient noise from one caused by infeasibility, and our within-block noise estimate is one way to draw that distinction.
Research Program
The first stage builds the evaluation harness. We take MO-Gymnasium environments from the MORL-Baselines suite: mo-hopper, mo-halfcheetah and minecart-v0, plus deep-sea-treasure as a small sanity case. Candidate sets come from the published GPI-LS, GPI-PD and PSL-MORL implementations. We then corrupt a chosen subset of reward channels with additive noise whose scale follows a time-varying envelope. The envelopes are taken from real non-stationary recordings, including DEMAND noise amplitude envelopes and bursty jackhammer and traffic profiles of the kind Wilson, Raj and Smaragdis studied. We add a stationary Gaussian control at matched average power. The noise can be shared across candidates within a block, or independent per episode, and both regimes are tested because the blocking trick only helps in the first.
The second stage implements the weighting rule and its operators on hesitant elements. Hesitant elements are built by quantizing each candidate's per-term episode returns into a small number of levels with empirical probabilities. The dual hesitant aggregation operators of Yu et al. are used for scoring, with pruning to a fixed number of levels to avoid the combinatorial growth that paper warns about. Their finding that averaging and geometric aggregation reverse rankings is treated as a nuisance variable, and we report both.
The third stage is an extension, and only if stage two succeeds. The same weights are used online as the scalarization inside PPO training on hand-designed reward terms, refreshed every block. The question is whether noise-driven swings in weight destabilize training less than entropy weighting does.
Baselines
The naive reference is equal weights across reward terms with plain mean scores, which is what most practitioners do when nobody can say what matters. It is run on exactly the same evaluation episodes. Beating it matters only if our rule also stays close to the clean-data ranking, since equal weights are trivially immune to noise-driven reweighting while ignoring discriminative power altogether.
The strongest existing method for the specific question of deriving unknown weights from hesitant evaluations is entropy weighting on the probabilistic dual hesitant matrix following Zhu, combined with Guo's credibility and power averaging from Guo. Both are computed on the same episodes with the same quantization. Neither paper reports numbers on any reinforcement learning or noisy benchmark: no reported result found on Lacuna for policy selection under non-stationary reward noise, and Zhu and Guo report only which alternative ranks first on synthetic matrices. We will therefore reproduce these methods ourselves. Beating them means a higher Kendall rank correlation with the clean-evaluation ranking and lower clean-utility regret of the selected policy, over at least ten seeds with non-overlapping confidence intervals, in the non-stationary condition.
The strongest engineering competitor is per-channel running mean and variance normalization followed by equal weighting, which is standard in reinforcement learning codebases. We tune its window length over the same grid as our forgetting factor.
For the multi-objective route, the reference is GPI-LS and PSL-MORL's expected utility under a uniform prior over weights. For context, PSL-MORL reports a hypervolume of $8.63 \times 10^6$ on MO-Ant-v2 against $7.48 \times 10^6$ for PD-MORL (Liu et al.). That number is on clean rewards and is a front-quality metric rather than a selection metric, so it situates our candidate generators rather than serving as the target.
Two ablations isolate the key ingredient. The first uses our rule with a single global, stationary noise estimate instead of block-local tracking, which tests whether non-stationary tracking matters. The second uses the rule with scalar means instead of hesitant elements, meaning a plain between-over-within ratio. That tests whether the fuzzy representation contributes anything beyond an F-statistic. Every method gets the same episode budget and the same hyperparameter grid size.
Why not just…?
Why not just normalize each reward term by its running standard deviation? This is the most serious threat. Normalization does shrink a noisy channel's raw scale. But it rescales the channel to unit variance, which is mostly noise, so the channel still contributes as much as a clean one. It also cannot separate between-candidate spread from within-candidate spread, because the running variance mixes the two. If normalization matches our rule on rank correlation in the shared-noise regime, the proposal's contribution shrinks to the blocking schedule alone. The head-to-head comparison is designed to show which outcome holds.
Why not just smooth rewards with a moving average or Wiener filter first? Smoothing assumes stationary noise statistics. Wilson et al. report roughly 0.5 PESQ gains over the Aurora Wiener filter on bursty noise. Fang et al. report a supervised denoiser falling to 2.93 dB SI-SDR on unseen noise, against 11.88 dB for a noise-aware alternative. Those results come from audio, though, and a reward stream is a much simpler signal. We therefore include an exponential-moving-average-then-entropy baseline with a tuned window rather than assume it fails.
Why not just learn the whole Pareto front and let a human pick? That approach avoids choosing weights during training, not choosing a policy. A human picking under noisy evaluations faces the same corrupted numbers. Roijers also notes that front-based methods scale poorly with the number of objectives. We use fronts as candidate sets, so this alternative is complementary to the proposal rather than competing with it.
Why not just use scalar between/within ratios and drop the fuzzy sets? This is a real possibility, and the second ablation tests it directly. If scalar ratios match hesitant elements, we will report that the hesitant representation is unnecessary for this purpose. The only case where we expect it to help is multimodal episode returns, for example policies that sometimes fall. In that case a mean-and-variance summary misrepresents the candidate, and Yu et al.'s rank reversals between aggregation operators suggest the shape of the distribution matters.
Decisive Checks
The first check is whether the failure we target exists at all. On mo-hopper and minecart-v0 with one channel under DEMAND-envelope noise, we measure the weight entropy methods assign to the corrupted channel as noise power increases. If that weight does not grow, the motivating failure is absent and the project stops.
The second check is the main result. We plot Kendall rank correlation with the clean ranking, and clean-utility regret of the selected policy, against noise power, separately for stationary and non-stationary noise and for shared and independent noise. The prediction is that our rule's advantage over the stationary-estimate ablation appears only under non-stationary shared noise. An advantage that appeared under stationary noise too would mean the gain comes from something other than tracking.
The third check is the online extension. Under the same injected noise, we compare PPO returns on clean reward terms and the variance of the weights over training, using our weights versus entropy weights.
References
- On Obtaining Stable Rankings
- Fuzzy Multi-Attribute Decision Making Method of Energy Alternative Evaluation
- The Probabilistic Dual Hesitant Fuzzy Multi-Attribute Decision-Making Method Based on Cumulative Prospect Theory and Its Application
- Discriminative training of variable-parameter HMMs for noise robust speech recognition
- Study on the Selection of Pharmaceutical E-Commerce Platform Considering Bounded Rationality under Probabilistic Hesitant Fuzzy Environment
- Relative Entropy Sorting Method Based on the Preference Information of Alternative
- Inverse Reward Design
- Efficient Methods for Multi-Objective Decision-Theoretic Planning
- Pareto Set Learning for Multi-Objective Reinforcement Learning
- Efficient Discovery of Pareto Front for Multi-Objective Reinforcement Learning
- Adaptive Reward Design for Reinforcement Learning
- A Toolkit for Reliable Benchmarking and Research in Multi-Objective Reinforcement Learning
- Regularized non-negative matrix factorization with temporal dependencies for speech denoising
- DUAL HESITANT FUZZY AGGREGATION OPERATORS
- Variational Autoencoder for Speech Enhancement with a Noise-Aware Encoder