Abstract
In high-volume production environments like Gmail, malware detection relies on efficient multi-stage pipelines. This paper investigates a critical vulnerability in the "routing" phase of such systems, where an ML model (Magika) identifies file types to determine which specialized scanner to use. Researchers from Google DeepMind and Google demonstrate that by modifying as few as 13 bytes, attackers can trick the system into misrouting malicious files, effectively bypassing detection. To counter this, the team developed and deployed a novel defense using a single round of AES encryption as a preprocessing step to disrupt adversarial gradients.
The System-Level Vulnerability
Google’s production malware detection pipeline uses a three-phase approach to maintain efficiency: signature-based scanning, routing via the Magika ML model, and specialized deep scanning. The security of this entire pipeline depends on the routing phase. If an attacker crafts an adversarial example that causes Magika to misidentify a malicious PDF as a benign text file, the file is sent to the wrong specialized scanner, which will find nothing, allowing the malware to reach the user's inbox.
Because Magika is open-source, attackers can study its parameters to craft "transferable" attacks that work against the production system. This highlights how ML components within a larger system can create unexpected and critical vulnerabilities.
Methodology: Crafting Format-Preserving Attacks
The researchers adapted the Greedy Coordinate Gradient (GCG) attack—originally for LLMs—to the domain of binary files. To ensure the modified files remained functional, they used a tool called Mitra to identify "blind spots" in file formats (such as metadata or comments) where bytes could be changed without breaking the file's malicious payload. To simulate a realistic black-box attack, they trained an ensemble of substitute models and optimized their attack to fool the entire ensemble simultaneously.
Experimental Results
The attacks proved highly effective across a wide range of file types. In a white-box setting, the researchers achieved a 90% success rate in misclassifying files by changing just 5 bytes. In more realistic scenarios, modifying roughly 13 bytes allowed them to successfully misroute malware across all tested formats, including PDF, ZIP, and Docx, while keeping the files functional.
Defense: Cryptographic Preprocessing
To mitigate these attacks, the authors proposed a novel defense inspired by cryptography. Before the ML model processes the file bytes, the system performs a single round of AES encryption using a secret key. Due to the high diffusion property of AES, small input changes lead to large, chaotic output changes. This creates a non-linear gradient landscape that makes it nearly impossible for an attacker to calculate a successful adversarial perturbation without the secret key. This defense was found to be more effective than traditional adversarial training and has been successfully deployed in production for the Gmail classifier.
Conclusion
This research demonstrates that the "router" in a detection pipeline is often the weakest link. While open-sourcing tools like Magika benefits the community, it also provides a roadmap for attackers. The success of the AES-based defense suggests that hybridizing cryptographic techniques with machine learning can provide robust and computationally efficient protection against adversarial threats in production environments.
Summary updated