RaPP: Novelty Detection with Reconstruction along Projection Pathway

Ki Hyun Kim, Sangwoo Shim, Yongsub Lim, Jongseob Jeon, Jeongwoo Choi, Byungchan Kim, Andre S. YoonICLR 2020

Novelty detection—identifying data points that differ significantly from a "normal" training set—is a critical task across various domains, from fraud detection and medical diagnosis to industrial equipment monitoring. Presented at ICLR 2020, RaPP (Reconstruction along Projection Pathway) introduces a clever extension to the classic autoencoder-based approach for novelty detection. By looking "under the hood" at hidden layers, the authors significantly improve the ability of neural networks to spot anomalies without requiring specialized training architectures.

The Problem: Beyond Input Space Reconstruction

The standard deep learning approach to novelty detection relies on an Autoencoder (AE). The AE is trained to compress and then reconstruct normal data. When a "novel" or anomalous sample arrives, the AE typically fails to reconstruct it accurately because it has never seen such data. The resulting "reconstruction error"—the difference between the input and the output—serves as the novelty score.

However, there is a significant gap in this standard method: it only measures reconstruction quality in the original input space. Deep networks learn a hierarchy of features. An anomaly might look somewhat normal in the raw input space but appear highly unusual in the abstract, high-level feature spaces of the hidden layers. Standard AEs ignore this rich internal information.

The RaPP Method: Peeking Under the Hood

To exploit these hidden layers, the authors propose a simple yet highly effective procedure:

  1. The Forward Pass: Feed an input $x$ into a trained autoencoder to get its reconstruction $\hat{x}$. During this process, record the activations $h_i(x)$ at every hidden layer $i$.
  2. The "Hidden Reconstruction": Feed the reconstructed output $\hat{x}$ back into the same encoder. Record the new activations $h_i(\hat{x})$.
  3. Comparison: Compare $h_i(x)$ and $h_i(\hat{x})$ at every layer.
The RaPP architecture compares the hidden activations of the original input with the hidden activations of its reconstruction.

The core mathematical insight is that $h_i(\hat{x})$ is effectively a "reconstruction" of the hidden features $h_i(x)$. If $x$ is normal, these two sets of activations should be very similar across all layers. If $x$ is novel, the discrepancies will compound along the "projection pathway."

The authors propose two ways to aggregate these layer-wise differences into a single novelty score:

  • SAP (Simple Aggregation along Pathway): A straightforward sum of squared Euclidean distances across all layers.
  • NAP (Normalized Aggregation along Pathway): A more robust metric that uses Singular Value Decomposition (SVD) on the training data to decorrelate neurons and scale the errors. This acts essentially as a Mahalanobis distance applied across the concatenated hidden spaces.

Key Insights and Advantages

  • No Special Architecture Required: The authors proved that you don't need a specialized network to find "hidden reconstructions." Simply passing the reconstructed input back through the existing encoder provides the necessary values. This means RaPP can be applied to almost any existing pre-trained autoencoder.
  • Hierarchical Sensitivity: By checking every layer, RaPP can detect anomalies that might fool the final output layer but look "wrong" to intermediate feature detectors.
  • Normalization is Key: The NAP metric is significantly more effective than SAP. It prevents layers with larger activation magnitudes or highly correlated neurons from dominating the novelty score, leading to much cleaner detection signals.

Datasets and Key Results

The researchers evaluated RaPP across a wide variety of data types, including tabular/industrial datasets (like CNC milling, storage system failures, and robotics malfunctions) and classic image benchmarks (MNIST and Fashion-MNIST). They used both "multimodal" (one class is novel, the rest are normal) and "unimodal" (one class is normal, the rest are novel) setups to ensure robustness.

Performance comparison showing NAP consistently outperforming standard reconstruction (Recon) and SAP across various autoencoder architectures.

Performance Boost: RaPP (specifically the NAP variant) consistently outperformed standard reconstruction error across nearly all 15 test scenarios. Furthermore, when compared against other modern novelty detection methods (like Deep SVDD, OCNN, and GAN-based approaches), RaPP applied to a Variational Autoencoder (RaPP-VAE) achieved state-of-the-art AUROC scores on both MNIST and Fashion-MNIST. The authors also demonstrated that performance generally improves as more hidden layers are included in the calculation, confirming the value of the projection pathway.

Limitations and Future Work

While highly effective, RaPP does come with a few trade-offs:

  • Computational Cost: RaPP requires a second pass through the encoder for every test sample. While neural network inference is generally fast, this effectively doubles the latency compared to standard reconstruction-based detection.
  • Metric Refinement: While NAP is a strong general-purpose metric, domain-specific knowledge could potentially be used to design even better aggregation functions for specific types of data.
  • Manifold Dependency: RaPP relies on the autoencoder successfully learning a representative "manifold" of normal data. If the base autoencoder is poorly trained or its bottleneck is improperly sized, RaPP's effectiveness will be inherently limited.

Summary updated

BibTeX
@inproceedings{kim2020rapp,
  author = {Ki Hyun Kim and Sangwoo Shim and Yongsub Lim and Jongseob Jeon and Jeongwoo Choi and Byungchan Kim and Andre S. Yoon},
  title = {{RaPP: Novelty Detection with Reconstruction along Projection Pathway}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
}
Figure