Adversarial Manipulation of Data Attribution Mechanisms

19 concepts13 papers1 proposal

When multiple participants pool their data to train a machine learning model—whether in a centralized data marketplace or a decentralized framework like Federated Learning (FL)—we face a fundamental question: How much is each participant's data worth?

To sustain collaboration, platforms use data valuation and attribution metrics to distribute financial rewards or allocate high-quality models. However, a major security flaw underpins almost all standard valuation techniques. Traditional metrics assume that participants are honest. In reality, strategic actors can exploit the mathematical formulations of these metrics to artificially inflate their perceived contribution, gaming the reward system without providing any real utility.


The Fragility of Data Attribution

To understand why data valuation is vulnerable, we must look at how attribution metrics compute value. Most frameworks rely on one of three paradigms:

  1. Game-Theoretic Semivalues (e.g., Shapley Value): Measuring a participant's marginal contribution across various subsets of all participants' data.
  2. Influence Functions: Using a model's gradients and Hessian matrix to approximate how a specific training point shifts the model's predictions.
  3. Memorization Scores: Tracking how deeply a model must memorize a specific sample to achieve its training objective.

As surveyed by Cong et al. in Data Pricing in Machine Learning Pipelines, these systems are highly susceptible to strategic manipulation, such as coalition formation or submitting duplicate data to claim multiple rewards.

The core vulnerability stems from a lack of adversarial robustness. In a comprehensive evaluation of data valuation algorithms, Jiang et al. demonstrated in OpenDataVal: a Unified Benchmark for Data Valuation that existing algorithms consistently fail to defend against valuation gaming, where malicious actors deliberately attempt to inflate the perceived value of low-quality or noisy samples.


How Adversaries Game the Metrics

Attackers exploit the fact that data attribution metrics are highly sensitive to the geometry of the model's loss landscape. By making subtle adjustments to their data or the model itself, adversaries can trigger massive spikes in their calculated value.

1. Manipulating the Loss Landscape (Influence Functions)

Influence functions estimate a data point's value by calculating how the loss changes if that point is removed. However, this calculation depends entirely on the model's parameters.

As proven by Yadav et al. in Influence-based Attributions can be Manipulated, an adversary can train a "malicious" model that achieves identical test accuracy to an honest model but possesses a radically altered gradient landscape. By subtly shifting the weights, the adversary forces the influence function to output massive scores for their chosen data points, allowing them to claim unfair financial rewards.

Similarly, memorization scores are not static properties of data; they depend on the input space's mathematical structure. In Efficiently Attacking Memorization Scores, Do et al. showed that an attacker can exploit the pseudoinverse of input data to efficiently manipulate memorization scores, demonstrating that data influence estimation is fundamentally fragile.

2. Crafting Subtle Data Perturbations

Instead of altering the model, an attacker can alter the data points themselves. In Adversarial Attacks on Data Attribution, Wang et al. introduced two systematic attack strategies:

  • Shadow Attack: Modifying a target data point with imperceptible perturbations so that it mimics the gradient characteristics of highly valuable data.
  • Outlier Attack: Pushing data points into sparse regions of the feature space where the model's gradients are naturally larger, artificially inflating the sample's perceived importance.

The authors demonstrated that these attacks can inflate a participant's compensation by up to 643%, completely undermining trust in data marketplaces.

3. Trajectory Prediction in Federated Learning

In Federated Learning, a central server evaluates client contributions based on the model updates they submit. If a client can predict how the global model will evolve, they can game the system.

[Malicious Client] ───(Predicts Future Trajectory)───► [Crafts Deceptive Update]
                                                               │
[Server Evaluates Contribution] ◄───(High Reward Assigned)─────┘

This strategy was realized by Xu et al. in ACE: A Model Poisoning Attack on Contribution Evaluation Methods in Federated Learning. Their framework, ACE, allows a malicious client to look ahead at the global model's optimization path. The client then crafts a low-quality or poisoned update that aligns perfectly with that future trajectory. The server misinterprets this alignment as a high-value contribution, rewarding the attacker while degrading the global model.


The Vulnerability of Decentralized and Fair Frameworks

The threat of reward manipulation extends across various collaborative learning paradigms, exposing structural weaknesses in both centralized and decentralized designs.

Shapley-Based Volatility

Even when frameworks attempt to secure the evaluation process, the underlying metrics remain unstable. For instance, Ma et al. developed a blockchain-based framework in Transparent Contribution Evaluation for Secure Federated Learning on Blockchain to compute Group-based Shapley Values securely on-chain. Yet, the authors noted that while this protects against passive curiosity, it does not prevent active adversaries from intentionally manipulating the group-value calculations.

Furthermore, Geimer et al. proved in On the Volatility of Shapley-Based Contribution Metrics in Federated Learning that Shapley-based rewards are highly volatile and shift drastically depending on the server's aggregation strategy, leaving an open question as to how these metrics behave under deliberate data poisoning.

Dynamic Weighting and Peer Feedback

Other frameworks attempt to dynamically adjust participant weights to maximize fairness or efficiency:

  • Reinforcement Learning Plugins: In A Fair Federated Learning Framework With Reinforcement Learning, Sun et al. proposed using policy gradient agents to distribute aggregation weights fairly. However, the authors highlighted that malicious participants can game these dynamic reward functions to claim higher weights without contributing diverse data.
  • Decentralized Peer Feedback: In robotic fleets or peer-to-peer networks, robots trade data based on localized utility scores. As explored by Geng et al. in Decentralized Sharing and Valuation of Fleet Robotic Data, without a central authority, these systems are highly vulnerable to peers providing falsified utility scores or injecting poisoned data to manipulate the collective learning process.
  • Slimmable Network Rewards: Frameworks like Aequa, introduced by Tastan et al. in Aequa: Fair Model Rewards in Collaborative Learning via Slimmable Networks, reward participants by giving them different widths of a slimmable neural network based on their contribution. While effective at stopping passive free-riders, the authors acknowledged that future work must address active adversaries attempting to manipulate the underlying contribution assessment.

Designing Strategy-Proof Defenses

To counter these vulnerabilities, researchers are shifting away from naive post-hoc metrics toward strategy-proof mechanisms, where honesty is mathematically guaranteed to be the optimal strategy.

In Incentives in Private Collaborative Machine Learning, Sim et al. detailed how standard collaborative data valuation systems collapse under data poisoning. To resolve this, the authors expanded their approach in Incentivizing Truthfulness and Collaborative Fairness in Bayesian Learning.

[Participant Data] ───► [Game-Theoretic Semivalues] ──┐
                                                      ▼
                        [Hidden Validation Function] ──► [Truthful Reward Allocation]

By combining game-theoretic semivalues with a hidden validation-based valuation function, the authors constructed a mechanism for Bayesian collaborative learning where participants cannot gain an advantage by duplicating data or injecting noise. Because the validation criteria remain hidden from the participants, any attempt to manipulate or perturb their data updates risks lowering their alignment with the hidden validation set, proving mathematically that truthfulness is the dominant strategy for maximizing rewards.

As machine learning pipelines become increasingly decentralized and commercialized, the field is moving away from assuming participant honesty. Securing the next generation of collaborative AI requires treating data valuation not just as an optimization problem, but as an adversarial frontier.

Key Foundational Work

Incentive Mechanisms for Federated Learning: From Economic and Game Theoretic Perspective (2021) — contributed 1 concept elements to this direction.
Transparent Contribution Evaluation for Secure Federated Learning on Blockchain (2021) — contributed 1 concept elements to this direction.
Data Pricing in Machine Learning Pipelines (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_data_valuation_manipulation_9529e701,
  title = {data valuation manipulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-valuation-manipulation/rdr_9529e70182924867847d73bc74deb46a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_data_valuation_manipulation_9529e701_v1,
  title = {data valuation manipulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-valuation-manipulation/rdr_9529e70182924867847d73bc74deb46a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake