TL;DR
This paper introduces a comprehensive framework for generating and evaluating adversarial attacks on tabular data, focusing on maintaining "coherence"—the logical consistency between interdependent features. By proposing a regression-based perturbation technique and a novel Class-Specific Anomaly Detection (CSAD) approach, the authors demonstrate that traditional metrics like $L_p$ norms fail to capture the true detectability of tabular attacks, while class-aware defenses can significantly improve model security.
What Problem Does This Paper Address?
Adversarial attacks are well-studied in images, where "imperceptibility" is defined by small pixel changes. However, tabular data (like credit scores or medical records) presents a unique challenge: features are heterogeneous and often strictly interdependent. For example, if an attacker increases a patient's "weight" feature to trigger a misclassification but leaves the "Body Mass Index (BMI)" feature unchanged, the resulting sample is mathematically impossible and easily detected.
Current research suffers from three major gaps:
- Lack of Coherence: Most attacks don't preserve complex feature dependencies.
- Inadequate Evaluation: Standard metrics like success rate and $L_p$ magnitude don't account for how "realistic" or "detectable" a sample is in a structured domain.
- Limited Comparative Analysis: There is no systematic comparison between query-based attacks (which need many model interactions) and transferability-based attacks (which use surrogate models) specifically for tabular data.
Methods
The authors propose a methodological framework to bridge these gaps, as illustrated in the overview below:
Key components include:
- Regression-based Perturbation: To maintain coherence, the framework identifies "dependent" features. When an "influencing" feature is changed, a regression model (trained on benign data) predicts the new value for the dependent feature, ensuring the relationship remains realistic.
- Class-Specific Anomaly Detection (CSAD): Traditional anomaly detection compares a sample to the entire dataset. CSAD instead compares the adversarial sample to the distribution of its predicted class. This is more sensitive because a sample might look "normal" globally but highly anomalous for the specific class it is trying to impersonate.
- Model Interpretation Stability: The authors use SHAP (SHapley Additive exPlanations) to see if an attack causes the model to rely on features in a way that is inconsistent with how it treats benign samples of that class. If the feature importance shifts wildly, the attack is flagged as inconsistent.
Datasets
The framework was evaluated using three diverse tabular datasets:
- Hateful Users on Twitter (Hate): 4,971 records, 115 features. Used to detect hateful vs. non-hateful users.
- Intensive Care Unit (ICU): 83,978 records, 74 features. Used to predict patient mortality. This dataset contains clear dependencies (like BMI vs. height/weight).
- Video Transmission Quality (VideoTQ): A proprietary dataset of 54,825 streaming broadcasts used to predict resolution quality.
Key Insights and Ideas
- The $L_2$ Fallacy: In tabular data, a small $L_2$ distance does not guarantee an "imperceptible" attack. A tiny change to a critical, highly-constrained feature can be more detectable than a large change to an unconstrained one.
- Query vs. Transferability Trade-off: Query-based attacks (like Boundary or HopSkipJump) achieve near-perfect success rates but require thousands of queries and often produce large, easily detectable perturbations. Transferability attacks are much harder to detect and require only one query, but they struggle with lower success rates on complex datasets.
- Class-Awareness is Critical: CSAD proved significantly more effective than global anomaly detection. By focusing on the target class distribution, it catches subtle inconsistencies that global models miss.
- Internal vs. External Consistency: An attack might look "coherent" in the feature space (external) but still be detected because it forces the model into an unstable decision-making state (internal), which SHAP-based metrics can uncover.
Key Results
- Detection Performance: CSAD often achieved detection rates twice as high as standard global approaches. For instance, on the VideoTQ dataset with a Random Forest model, CSAD detected 66-87% of query-based attacks that global detection missed entirely (0% detection).
- Attacker Effort: While query-based attacks are highly effective (98%+ success), they are computationally expensive. The authors found that transferability-based attacks, while less successful (often ~50% or lower on ICU), were significantly faster to generate once a surrogate model was trained.
- Feature Importance: SHAP-based anomaly detection revealed that even "successful" adversarial samples often exhibit anomalous feature importance values, providing a robust second line of defense.
Limitations and Future Work
The authors identify several areas for further exploration:
- Circular Dependencies: The current regression-based adjustment assumes a directed dependency (A influences B). Handling mutual or circular dependencies (where A and B influence each other) remains an open challenge.
- Latent Dependencies: The framework currently relies on domain knowledge or preliminary analysis to identify feature relationships. Automating the discovery of these "hidden" constraints would make the attacks more robust.
- Multi-class Generalization: While the framework is designed to support multi-class settings, the empirical evaluation focused primarily on binary classification.
For those interested in reproducing these results or building upon the framework, the authors have made their code and datasets available on GitHub.
Summary updated