Abstract
Adversarial machine learning research has historically focused on homogeneous data types like images and text. However, tabular data—prevalent in domains such as finance and cybersecurity—presents unique challenges due to its heterogeneous nature and strict feature constraints. This paper introduces the Feature Importance Guided Attack (FIGA), a model-agnostic, gradient-free adversarial attack specifically designed for tabular data. FIGA identifies and perturbs influential features toward the average values of a target class. Our evaluation shows that FIGA achieves a 94% success rate in feature space and successfully generates functional, deceptive phishing websites in the problem space, highlighting a significant vulnerability in tabular data classifiers.
Introduction
Tabular data is the backbone of many critical machine learning applications, including credit scoring, medical diagnosis, and phishing detection. Unlike image data, where features (pixels) are homogeneous and bounded, tabular data is "heterogeneous," comprising a mix of integers, categories, and floating-point numbers. Furthermore, these features often have strict constraints and interdependencies; for instance, an individual's age cannot be negative, and adding an image to a website inherently alters other features like HTML tag counts.
This paper addresses the inverse feature-mapping problem: the challenge of translating a mathematical perturbation in feature space into a functional, real-world object (e.g., a phishing site) that remains deceptive and operational.
Methodology
The authors propose FIGA, a two-step, gradient-free attack strategy that does not require knowledge of the target model's architecture or gradients.
- Ranking & Direction: FIGA first ranks features by importance using standard methods such as Gini Impurity or Information Gain. It then determines the "direction" of the attack by comparing the input sample to the mean values of the target class.
- Perturbation: The algorithm perturbs the top $n$ features by a total amount $\epsilon$. These changes are scaled to remain within the known min/max bounds of the training data, ensuring the adversarial sample remains "valid."
Experimental Setup
The study evaluates FIGA across several diverse datasets:
- Primary Phishing Dataset: A custom-crawled set of 348,739 URLs and their associated HTML source code.
- Benchmark Phishing Datasets: Three established benchmarks (Kang, Mamun, and Lee) to test robustness.
- Adult Dataset: A financial dataset from the UCI repository used to demonstrate generalizability to other tabular domains.
Results and Discussion
Feature Space Performance
In the mathematical feature space, FIGA achieved an average 94% success rate across all datasets and models. It reached a 100% success rate against non-tree models, such as Logistic Regression and FastAI Tabular, even with minimal perturbations.
Key Insights
- The "Mimicry" Intuition: Fooling a classifier can be achieved by moving the most important features of a "bad" sample toward the average values of a "good" sample.
- Robustness of Tree-Based Models: Tree-based models (Random Forest, XGBoost) proved significantly more robust than neural networks (MLPs) or linear models, requiring larger perturbations to fail.
- The Importance of epsilon: Among the tunable parameters ($n$ features, $\epsilon$ perturbation, and ranking method), the total perturbation amount ($\epsilon$) was the most critical factor in determining attack success.
Problem Space Feasibility
The authors generated 10,000 actual adversarial phishing websites. While these sites were visually identical to the originals, they successfully evaded detection with a 13.05% average success rate. The drop from feature space success is attributed to "side effects"—unintended changes to features that occur when modifying a real-world object.
Conclusion
FIGA demonstrates that model-agnostic attacks can effectively compromise tabular data classifiers. While highly successful in feature space, the transition to the problem space remains challenging due to feature interdependencies. Future work will investigate iterative versions of FIGA and "zero-knowledge" scenarios where dataset statistics must be estimated through sampling.
Summary updated