SecureDL: Securing Code Execution and Access Control for Distributed Data Analytics Platforms

Fahad Shaon, Sazzadur Rahaman, Murat Kantarcioglu

Abstract

Distributed data analytics platforms like Apache Spark are designed for high-performance scale, often at the expense of fine-grained security. Existing security solutions, typically based on Inline Reference Monitors (IRM), are vulnerable to "transient evasion attacks"—stealthy exploits where users abuse internal platform APIs to bypass security controls. This paper introduces SecureDL, a defense-in-depth framework that combines proactive static analysis with reactive runtime sandboxing. SecureDL screens user code for suspicious API usage before execution and enforces runtime checks via a hardened Java Security Manager and binary integrity verification. Evaluation on TPC-H and HiBench benchmarks demonstrates that SecureDL effectively blocks evasion attacks with a modest performance overhead of approximately 4%.

Introduction

As distributed analytics platforms become central to enterprise data processing, the need for robust access control has intensified. However, the authors identify a critical flaw in current security models: transient evasion attacks. In these scenarios, a malicious insider uses legitimate platform APIs (such as Java Reflection or internal package calls) to access raw data before security filters are applied. Because these operations occur entirely within the memory of the running job, they leave no traces in system logs. Standard sandboxing techniques, such as the default Java Security Manager, are often insufficient to prevent these sophisticated workarounds.

The SecureDL Framework

SecureDL addresses these vulnerabilities through a two-layered defense strategy that integrates into the existing Spark ecosystem.

Proactive Defense

Before a job is executed, SecureDL performs a "pre-flight check" using static program analysis. It screens user-submitted code for blockable signatures, such as attempts to declare classes within protected internal packages (e.g., org.apache.spark). Additionally, it employs "soundy" dataflow analysis to detect suspicious but non-blockable API usage, such as using reflection to traverse an RDD’s internal pointers to access unauthorized data.

Reactive Defense

To catch threats that might bypass static analysis, SecureDL employs a runtime reactive layer. This includes binary integrity checking to ensure the Trusted Computing Base (TCB) has not been tampered with, and a hardened Java Security Manager to block unauthorized system calls. Furthermore, runtime instrumentation injects security checks that trigger exceptions if code attempts to invoke sensitive APIs on protected data objects during execution.

Architecture of the SecureDL framework, illustrating the interaction between the Policy Dispatcher, proactive/reactive agents, and the Spark cluster.

Experimental Evaluation

The authors evaluated SecureDL using industry-standard benchmarks and real-world data:

  • Performance Overhead: Testing on a 6-node Hadoop cluster using TPC-H queries showed an average overhead of only 4%. While complex queries involving multiple joins saw higher overhead (up to 16%), simpler aggregation tasks remained highly efficient.
  • Security Effectiveness: In a test suite of 15 custom-crafted transient evasion attacks, the proactive analysis successfully detected and blocked every attempt.
  • Real-world Compatibility: The authors analyzed 2,120 real-world Spark repositories from GitHub. Of the analyzable jars, the system exhibited a very low false-positive rate, only flagging highly unusual developer behaviors such as manual external network connections or internal package redefinitions.

Conclusion

SecureDL provides a pragmatic and effective solution for securing distributed analytics platforms against sophisticated API-based attacks. By combining the gatekeeping capabilities of static analysis with the bodyguard-like protection of runtime sandboxing, it achieves high security without sacrificing the performance that makes platforms like Spark valuable. Future work aims to extend these protections to non-JVM interfaces such as PySpark and R.

Summary updated

Abstract

Distributed data analytics platforms (i.e., Apache Spark, Hadoop) enable cost-effective storage and processing by distributing data and computation to multiple nodes. Since these frameworks' design was primarily motivated by performance and usability, most were assumed to operate in non-malicious settings. Hence, they allow users to execute arbitrary code to analyze the data. To make the situation worse, they do not support fine-grained access control inherently or offer any plugin mechanism to enable it - which makes them risky to be used in multi-tier organizational settings. There have been attempts to build "add-on" solutions to enable fine-grained access control for distributed data analytics platforms. In this paper, we show that by knowing the nature of the solution, an attacker can evade the access control by maliciously using the platform-provided APIs. Specifically, we crafted several attack vectors to evade such solutions. Next, we systematically analyze the threats and potentially risky APIs and propose a two-layered (i.e., proactive and reactive) defense to protect against those attacks. Our proactive security layer utilizes state-of-the-art program analysis to detect potentially malicious user code. The reactive security layer consists of binary integrity checking, instrumentation-based runtime checks, and sandboxed execution. Finally, Using this solution, we provide a secure implementation of a new framework-agnostic fine-grained attribute-based access control framework named SecureDL for Apache Spark. To the best of our knowledge, this is the first work that provides secure fine-grained attribute-based access control distributed data analytics platforms that allow arbitrary code execution. Performance evaluation showed that the overhead due to added security is low.
BibTeX
@misc{shaon2021securedl,
  author = {Fahad Shaon and Sazzadur Rahaman and Murat Kantarcioglu},
  title = {{SecureDL: Securing Code Execution and Access Control for Distributed Data Analytics Platforms}},
  year = {2021},
  eprint = {2106.13123},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.13123},
}
Figure