Detection under Privileged Information.

Z. Berkay Celik, Patrick D. McDaniel, Rauf Izmailov, Nicolas Papernot, Ryan Sheatsley, Raquel Alvarez, Ananthram SwamiAsiaCCS 2018 · DOI · OpenReview

TL;DR

This paper introduces "privileged information" – data available during training but not at runtime – to significantly improve the accuracy and resilience of detection systems, particularly in cybersecurity. By adapting techniques like knowledge transfer, model influence, and distillation, the authors demonstrate substantial gains in detecting threats like malware and botnets, and even in face recognition.

What Problem Does This Paper Address?

Traditional detection systems, especially those using machine learning, are limited by the features they can access at runtime. This means valuable information that might be available during the initial training phase (e.g., from forensic investigations, detailed logs, or complex analyses) is often discarded because it's too computationally expensive, time-consuming, or simply unavailable when a real-time decision needs to be made. This limitation hinders the ability of these systems to accurately identify sophisticated threats. For instance, after a major data breach, the wealth of forensic data collected might reveal advanced persistent threats, but this information cannot be directly used to train the existing real-time detection systems. The paper aims to bridge this gap by enabling detection models to leverage this "privileged" information during training to build more robust and accurate systems without increasing runtime costs.

Methods

The core contribution of this paper lies in adapting and extending three machine learning techniques to incorporate privileged information into detection models:

  • Knowledge Transfer: This approach aims to estimate privileged features from the standard, runtime-available features. The authors propose using mapping functions (like regression or similarity-based methods) to learn this relationship. At runtime, the model uses the standard features and the estimated privileged features to make a prediction.
  • Model Influence: This method integrates privileged information by adding constraints to the training error of the detection model. Essentially, it uses privileged features to "correct" or influence the decision boundary learned from standard features, aiming for a more accurate separation of classes. The paper demonstrates this with an SVM-based approach, creating an "SVM+" model.
  • Distillation: Inspired by model compression techniques, distillation uses the "soft labels" (probability distributions over classes) generated by a model trained on privileged information to guide the training of a detection model that only uses standard features at runtime. This allows the detection model to learn the nuanced relationships captured by the privileged data.

The paper also introduces a method for feature engineering to select the most impactful privileged features by measuring their "accuracy gain" on hard-to-classify examples.

Datasets

The authors evaluate their proposed methods on four diverse security-relevant systems:

  • Face Authentication: Uses a dataset of facial images.
  • Fast-flux Bot Detection: Utilizes DNS requests from benign and active fast-flux servers.
  • Malware Traffic Detection: Employs network flow statistics from benign applications and Zeus botnet traffic.
  • Malware Classification: Leverages the Microsoft malware dataset, including binary content and metadata logs.

The paper does not introduce new benchmark datasets but rather applies their techniques to existing, real-world data.

Key Insights and Ideas

  • Privileged Information is a Game-Changer: The fundamental idea that data unavailable at runtime can significantly boost detection accuracy is a key takeaway. This challenges the long-held assumption that only runtime features are relevant for training.
  • No One-Size-Fits-All Approach: The paper demonstrates that different privileged information techniques (Knowledge Transfer, Model Influence, Distillation) perform better on different tasks. The choice depends on the domain, data characteristics, and desired trade-offs.
  • Feature Engineering is Crucial: Simply having privileged data isn't enough; intelligently selecting which privileged features to use is vital for maximizing gains and avoiding noise. The "accuracy gain" metric for hard-to-classify examples is a novel way to approach this.
  • Balancing Runtime Constraints and Accuracy: The core benefit is achieving higher accuracy without incurring additional runtime costs, which is critical for real-world deployment in resource-constrained environments.
  • Defense Against Adversarial Attacks: The paper suggests that privileged information can also serve as a defense mechanism, making machine learning models more robust against adversarial manipulations.

Key Results

The empirical evaluation shows significant improvements across all tested systems:

  • Face Authentication: Up to 16.9% relative decrease in detection error (using Distillation).
  • Fast-flux Bot Detection: Up to 7.7% relative decrease in detection error (using Knowledge Transfer).
  • Malware Traffic Detection: Up to 8.6% relative decrease in detection error (using Distillation).
  • Malware Classification: Up to 7.3% relative decrease in detection error (using Model Influence).

In general, the privileged information-augmented systems achieved accuracy, precision, and recall scores that were often close to or even matched those of a "complete set" model (which uses all features, including privileged ones, at runtime – an impractical scenario). The paper also highlights that Model Influence and Distillation introduce no runtime overhead, while Knowledge Transfer might introduce some delay depending on the mapping function used.

Limitations and Future Work

  • Model Dependency: While Knowledge Transfer is model-agnostic, Model Influence is presented with an SVM objective, and Distillation requires a softmax output layer. Adapting these to other models is an area for future work.
  • Detection Overhead: Knowledge Transfer, particularly with similarity-based mapping, can introduce runtime delays. The authors suggest stratified sampling and using regression-based mapping to mitigate this.
  • Model Optimization: Achieving optimal performance requires careful tuning of parameters for each approach, which can be time-consuming.
  • Training Overhead: Model Influence can be computationally intensive for large datasets or high-dimensional feature spaces.
  • Mapping Function Selection (Knowledge Transfer): The effectiveness of Knowledge Transfer heavily relies on choosing an appropriate mapping function. The paper notes that for malware classification, the chosen functions failed to find a good relation, leading to inconsistent results. Future work could explore more advanced mapping functions.
  • Adversarial Robustness: The authors suggest further exploring privileged information as a defense mechanism against adversarial attacks.
  • Broader Applications: The paper advocates for exploring new models and systems using privileged features to promote lightweight, accurate, and robust detection across various environments.

Summary updated

Abstract

For well over a quarter century, detection systems have been driven by models learned from input features collected from real or simulated environments. An artifact (e.g., network event, potential malware sample, suspicious email) is deemed malicious or non-malicious based on its similarity to the learned model at runtime. However, the training of the models has been historically limited to only those features available at runtime. In this paper, we consider an alternate learning approach that trains models using privileged information--features available at training time but not at runtime--to improve the accuracy and resilience of detection systems. In particular, we adapt and extend recent advances in knowledge transfer, model influence, and distillation to enable the use of forensic or other data unavailable at runtime in a range of security domains. An empirical evaluation shows that privileged information increases precision and recall over a system with no privileged information: we observe up to 7.7% relative decrease in detection error for fast-flux bot detection, 8.6% for malware traffic detection, 7.3% for malware classification, and 16.9% for face recognition. We explore the limitations and applications of different privileged information techniques in detection systems. Such techniques provide a new means for detection systems to learn from data that would otherwise not be available at runtime.
BibTeX
@inproceedings{celik2018detection,
  author = {Z. Berkay Celik and Patrick D. McDaniel and Rauf Izmailov and Nicolas Papernot and Ryan Sheatsley and Raquel Alvarez and Ananthram Swami},
  title = {{Detection under Privileged Information.}},
  booktitle = {AsiaCCS},
  year = {2018},
  doi = {10.1145/3196494.3196502},
}
Figure