Ten-Guard: Tensor Decomposition for Backdoor Attack Detection in Deep Neural Networks

Khondoker Murad Hossain, Tim Oates

In the modern AI landscape, few researchers train massive models from scratch. Instead, the standard operating procedure is to download a pre-trained model from a hub like Hugging Face and fine-tune it. But how do you know that the model you just downloaded doesn't contain a "backdoor"—a hidden trigger that causes it to misbehave only when it sees a specific, rare input?

A new paper from researchers at UMBC introduces TEN-GUARD, a novel pipeline that uses advanced tensor decomposition to detect these "trojaned" models with higher accuracy and significantly better speed than current state-of-the-art methods.

TL;DR

TEN-GUARD identifies backdoored neural networks by analyzing their internal activations using tensor decomposition methods like Independent Vector Analysis (IVA) and PARAFAC2. It outperforms existing detection tools in both accuracy and efficiency, offering a scalable way to vet third-party models before they are deployed in sensitive environments.

What Problem Does This Paper Address?

The "provenance" of deep learning models is a growing security concern. A malicious actor could release a high-performing model that functions perfectly on standard data but contains a "backdoor." For example, an autonomous driving model might work flawlessly until it sees a specific sticker on a stop sign, at which point it misclassifies it as a speed limit sign.

Detecting these backdoors is notoriously difficult because the "trigger" is unknown. Existing defensive methods often rely on "reverse-engineering" the trigger through expensive optimization processes (like Neural Cleanse) or making narrow assumptions about what the trigger looks like. These methods are often computationally heavy and struggle to scale as models and datasets grow.

Methods

TEN-GUARD shifts the focus from the weights of the model to its activations. The core idea is that backdoored models, when processing data, will exhibit unique, shared statistical patterns in their internal representations that clean models do not.

The pipeline works in three main steps:

  1. Feature Extraction: The researchers feed images into a set of pre-trained models and capture the activations from the final hidden layer. They use Random Projection (RP) to compress these high-dimensional activations into a uniform size, making the process memory-efficient and model-agnostic.
  2. Tensor Decomposition: They apply two specific techniques—Independent Vector Analysis (IVA) and Parallel Factor Analysis (PARAFAC2)—to the activation data. These methods are designed to find common "sources" or patterns across multiple datasets (in this case, multiple models).
  3. Detection: By analyzing the "mixing matrix" or calculating cross-correlation between models, the system can identify clusters. Trojaned models tend to reside in the same subspace because they react to triggers in a statistically similar way. If a test model shows significant correlation with known backdoored patterns, it is flagged.

Datasets

The authors evaluated TEN-GUARD across several benchmarks:

  • MNIST: 450 custom-trained CNNs (50% clean, 50% poisoned with a white patch trigger).
  • CIFAR-10: 550 ResNet-18 models.
  • NIST TrojAI (Round 1 & 2): A rigorous, large-scale competition dataset featuring complex architectures (ResNet50, DenseNet121, Inception-v3) and multi-class poisoning on synthetic traffic sign data.

Key Insights and Ideas

  • The Subspace Signal: The most critical "aha moment" is that trojaned models aren't just random outliers; they share a common "subspace" in their activation behavior. Tensor decomposition is uniquely suited to finding these shared latent structures across a population of models.
  • Model Agnosticism: Because the method uses Random Projection on final layer activations, it doesn't care about the specific architecture of the network. It treats the model as a black box that produces a signal.
  • Deterministic Advantage: The researchers found that PARAFAC2 generally outperformed IVA. This is because PARAFAC2 is more deterministic and robust to noise, and it doesn't require the same strict statistical assumptions that IVA does.

Key Results

TEN-GUARD (specifically the PARAFAC2 variant) demonstrated a clear edge over existing methods like Neural Cleanse (NC), ABS, and K-Arm:

  • Accuracy: On the difficult TrojAI ResNet50 dataset, TEN-GUARD achieved an ROC-AUC of 0.89, surpassing the closest competitor, K-Arm (0.88), and significantly beating earlier methods like NC (0.74).
  • Efficiency: This is where the paper shines. For the TrojAI datasets, TEN-GUARD processed the models in roughly 240 seconds. In contrast, optimization-based methods like ULP took nearly 4,000 seconds. That is a 15x speedup, making TEN-GUARD viable for real-time MLOps pipelines.
  • Clustering Clarity: Using Silhouette scores (a measure of how well-separated clusters are), the authors showed that their method creates very distinct groupings of "clean" vs. "trojaned" models, reducing the likelihood of false positives.

Limitations and Future Work

While highly effective, the authors noted some challenges:

  • Complexity of Triggers: Performance was slightly lower on the TrojAI datasets compared to MNIST. This is attributed to the "greater trigger variations" (size, color, location) and the increased depth of the models used in the NIST competition.
  • Layer Selection: The current study focused primarily on final layer activations. The authors suggest that exploring activations from multiple layers might provide an even stronger signal for detection.
  • Adaptive Attacks: As with all defensive measures, there is the risk of "adaptive attacks" where a malicious actor specifically trains a trojan to avoid creating the subspace patterns that TEN-GUARD looks for. Investigating the limits of this "cat-and-mouse" game is a natural next step.

Summary updated

Abstract

As deep neural networks and the datasets used to train them get larger, the default approach to integrating them into research and commercial projects is to download a pre-trained model and fine tune it. But these models can have uncertain provenance, opening up the possibility that they embed hidden malicious behavior such as trojans or backdoors, where small changes to an input (triggers) can cause the model to produce incorrect outputs (e.g., to misclassify). This paper introduces a novel approach to backdoor detection that uses two tensor decomposition methods applied to network activations. This has a number of advantages relative to existing detection methods, including the ability to analyze multiple models at the same time, working across a wide variety of network architectures, making no assumptions about the nature of triggers used to alter network behavior, and being computationally efficient. We provide a detailed description of the detection pipeline along with results on models trained on the MNIST digit dataset, CIFAR-10 dataset, and two difficult datasets from NIST’s TrojAI competition. These results show that our method detects backdoored networks more accurately and efficiently than current state-of-the-art methods.
BibTeX
@inproceedings{hossain2024ten,
  author = {Khondoker Murad Hossain and Tim Oates},
  title = {{Ten-Guard: Tensor Decomposition for Backdoor Attack Detection in Deep Neural Networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=qvyEjTPCID},
}

Loading related content...

Figure