Adversarial Manipulation of Internal Feature Representations

161 concepts110 papers1 proposal

Shifting the Battleground from Labels to Embeddings

Traditional adversarial attacks operate like a deceptive signpost. By adding imperceptible noise to an input image, an attacker forces a neural network's final layer to cross a decision boundary—turning a "panda" into a "gibbon" while leaving the human perception of the image completely unchanged. This paradigm, however, assumes that the system's primary output is a discrete classification label.

Modern deep learning architectures often operate differently. Systems for face recognition, image retrieval, vision-language alignment, and cooperative perception rely on continuous vector spaces. Instead of assigning a label, they map inputs into a high-dimensional embedding space where semantic similarity is measured by geometric distance (such as Euclidean distance or cosine similarity).

Because these systems rely on the geometric layout of the latent space, a new class of adversarial attacks has emerged: internal representation manipulation. Instead of optimizing against a final classification loss, these attacks directly manipulate the intermediate feature maps and latent vectors inside the network. By treating the internal layers as the primary target, adversaries can hijack a model's internal perception, execute task-agnostic disruptions, and achieve unprecedented black-box transferability.

Feature Mimicry and Latent Hijacking

The first core mechanism of representation manipulation is feature mimicry, where an attacker forces the internal activations of a source input to match those of a specific target input or class distribution.

In Adversarial Manipulation of Deep Representations, Sabour et al. introduced "feature adversaries." They demonstrated that an attacker can apply an imperceptible pixel-level perturbation to a source image that forces its intermediate layer representations to nearly perfectly match the activation vector of a completely different "guide" image. Within the deep layers of the network, the perturbed source image becomes indistinguishable from the guide image, effectively hijacking the network's internal perception.

Standard Input (Source) ----> [ Deep Layers ] ----> Vector A \
                                                              \-- Optimized to match!
Perturbed Input (Source) ---> [ Deep Layers ] ----> Vector B  /
                                                              /
Natural Input (Guide) -------> [ Deep Layers ] ----> Vector B /

This works because the high-dimensional activation spaces of deep networks are incredibly dense. Tiny, highly coordinated shifts in the input space can cause massive shifts in the latent space without introducing human-detectable artifacts.

This geometric vulnerability has severe implications for security architectures. For instance, open-set recognition models like Openmax attempt to detect adversarial or unknown inputs by monitoring their distance from known class averages (Mean Activation Vectors). However, Rozsa et al. introduced the Logits Optimized Targeting System (LOTS), demonstrating that minimizing the Euclidean distance between an input's penultimate activations and a target class's Mean Activation Vector allows adversarial examples to bypass these distance-based defenses entirely. The perturbed input mimics the expected internal distribution so precisely that it appears as a natural inlier.

This mechanism is particularly devastating for biometric verification. As shown by Massoli et al., optimizing an input to minimize the distance between its internal feature embeddings and a target identity's embedding allows an attacker to impersonate individuals in similarity-based face recognition systems far more effectively than traditional label-flipping noise.

Representation Disruption and Task-Agnostic Blindness

When an attacker does not have a specific target identity or category in mind, they can instead deploy representation disruption. Rather than guiding the latent vector toward a specific point, this strategy maximizes the geometric distance or divergence between the clean embedding and the perturbed embedding.

This approach is uniquely powerful against foundation models and multimodal systems, where a single backbone encoder feeds into multiple downstream tasks or user prompts. For example, the Segment Anything Model (SAM) uses a shared image encoder to generate a latent representation, which is then processed by various downstream decoders based on user-defined points, boxes, or text prompts.

Croce et al. demonstrated that by maximizing the distance between clean and perturbed embeddings within this latent space, an attacker can execute "prompt-agnostic" attacks. Because the core visual representation is corrupted at the encoder level, the model's downstream capabilities are completely neutralized, causing it to fail consistently regardless of the specific prompt provided by the user.

Clean Image ------> [ Image Encoder ] ----> Coherent Latent Space ----> Successful Downstream Tasks
                                                    |
                                            (Maximize Distance)
                                                    v
Perturbed Image --> [ Image Encoder ] ----> Corrupted Latent Space ----> Total Downstream Failure

The intuition here is straightforward: by destroying the semantic coherence of the embedding before it reaches downstream modules, the attack introduces a fundamental blindness. Naseer et al. leveraged this to generate highly transferable adversarial examples without requiring any class labels, simply by maximizing the distance between internal feature representations of clean and perturbed images within a fixed, pre-trained network.

This representation-space perturbation was also shown by Zhang et al. to successfully compromise Multimodal Large Language Models like Google's Bard. By shifting an image's latent embedding as far as possible from its original coordinates, the attack disrupts the model's internal understanding of the visual content, forcing object misidentification and bypassing internal safety filters without needing to target a specific textual output.

Why Feature-Space Attacks Transfer Across Models

One of the most compelling properties of internal representation manipulation is its high transferability in black-box scenarios. In a strict black-box setting, an attacker has no knowledge of the target model's architecture, parameters, or exact label space. Attacking the final classification layer of a surrogate model often fails to transfer because decision boundaries are highly architecture-specific.

However, feature-space attacks exploit a fundamental property of deep learning: representation convergence. As detailed by Gu, different neural network architectures trained on similar data tend to learn highly convergent internal representations of semantic concepts. A dog, a car, or a face will map to clusters that share similar relative geometric structures across different models.

When an attacker optimizes a perturbation to distort or hijack intermediate feature maps on a surrogate model, they are disrupting these shared semantic concepts rather than model-specific decision boundaries. Wu et al. utilized this by targeting the internal semantic representations of a self-supervised surrogate model. Because the attack focused on feature-level disruption rather than boundary-crossing, the resulting adversarial examples transferred successfully across diverse target domains, even when the target model's label space was entirely unknown.

Vulnerabilities in Distributed and Metric Systems

Manipulating internal representations allows adversaries to exploit systems where raw data is never exposed, or where decisions are entirely decentralized:

  • Image Retrieval Systems: Systems that rank search results based on vector proximity can be completely scrambled. Li et al. demonstrated that universal adversarial perturbations can disrupt the relative distances between images in a high-dimensional feature space, breaking neighborhood relationships and scrambling search results without needing to target specific image categories.
  • Distributed Split Learning: In privacy-preserving split learning, a client processes data through initial layers and sends the intermediate features to a server. Chen et al. showed that an adversary can optimize perturbations to minimize the cosine similarity between original and modified intermediate features. This forces the model into divergent internal states, degrading performance without requiring access to the final classification layers.
  • Cooperative Perception: In autonomous driving networks, vehicles share intermediate neural features (such as Bird's-Eye-View maps) to build a collective perception of the environment. As shown by Tu et al. and Wang et al., injecting subtle perturbations directly into these shared intermediate feature maps allows attackers to bypass non-differentiable preprocessing stages (like raw LiDAR point cloud manipulation). This directly corrupts the collective representation space, causing severe coordination failures or extreme processing latencies while remaining hidden from standard transmission encryption.

Key Foundational Work

Adversarial Robustness: Softmax versus Openmax (2017) — contributed 6 concept elements to this direction.
Adversarial Manipulation of Deep Representations (2016) — contributed 4 concept elements to this direction.
Fast Feature Fool: A data independent approach to universal adversarial perturbations (2017) — contributed 3 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_internal_representation_manipulation_bcf8d2f9,
  title = {internal representation manipulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/internal-representation-manipulation/rdr_bcf8d2f9a15c42a4be79169dd4cdd50a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_internal_representation_manipulation_bcf8d2f9_v1,
  title = {internal representation manipulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/internal-representation-manipulation/rdr_bcf8d2f9a15c42a4be79169dd4cdd50a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake