Learning Similarity Metrics with Twin Networks

634 concepts523 papers15 proposals

Standard classification networks excel at mapping an input to a discrete set of predefined class probabilities. However, this paradigm breaks down when the number of target categories is massive, dynamic, or entirely unknown at training time. For instance, verifying whether two signatures belong to the same individual cannot be framed as a simple 1-of-$N$ classification task, because new users join systems constantly, and a model cannot be retrained for every new person.

To solve this, Jane Bromley et al. introduced the Siamese neural network architecture in their foundational work, Signature Verification Using a Siamese Time Delay Neural Network. Instead of predicting a class label, the network learns a distance metric. It maps pairs of inputs into a continuous, low-dimensional embedding space where geometric proximity directly mirrors semantic or structural similarity.

The Intuition of Weight Sharing and Symmetry

The core mechanism behind a Siamese network is its use of twin subnetworks that share the exact same parameters (weights and biases).

Input A ───► [ Subnetwork (Weights: W) ] ───► Embedding A ───┐
                                                            ▼
                                                      Distance Metric ───► Similarity Score
                                                            ▲
Input B ───► [ Subnetwork (Weights: W) ] ───► Embedding B ───┘

When evaluating the similarity between two entities—whether they are source code fragments, images, or text strings—the comparison must be mathematically symmetric. The similarity of item $A$ to item $B$ must be identical to the similarity of item $B$ to item $A$. As Vaibhav Saini demonstrates in Oreo: Detection of Clones in the Twilight Zone, enforcing a shared-weight architecture guarantees this symmetry. Because both inputs pass through the exact same transformation function, the resulting vector space is perfectly consistent. This prevents the model from producing conflicting similarity scores based purely on the order in which inputs are presented. Furthermore, weight sharing slashes the number of parameters in half, which acts as a powerful regularizer and forces the network to learn generalized features rather than memorizing specific inputs.

Shaping the Embedding Space via Contrastive Learning

To construct a meaningful geometric space, Siamese networks rely on pair-based or triplet-based objective functions rather than standard cross-entropy loss.

The most common approach is contrastive loss. When presented with a pair of items known to be similar (a positive pair), the network minimizes the distance between their embeddings. Conversely, when presented with a dissimilar pair (a negative pair), the network maximizes their distance, often enforcing a minimum separation margin to prevent the model from wasting capacity pushing already-distant points further apart.

For example, Guoxian Dai et al. utilize contrastive loss in Siamese CNN-BiLSTM Architecture for 3D Shape Representation Learning to map complex 3D shape descriptors into a shared space for accurate retrieval. Similarly, Xiaojun Xu et al. use this contrastive mechanism in Neural Network-based Graph Embedding for Cross-Platform Binary Code Similarity Detection to identify functional equivalence in code across entirely different hardware architectures and compiler optimization levels.

To further refine these boundaries, architectures can expand to three branches. As explored by Yibo Hu et al. in Attention-Set Based Metric Learning for Video Face Recognition, processing an anchor, a positive sample, and a negative sample simultaneously allows the network to explicitly pull matching identities closer while pushing non-matching identities away, correcting for sample biases in complex datasets.

Bridging Heterogeneous Domains and Modalities

Because Siamese networks focus on learning relative metric spaces rather than absolute classifications, they excel at aligning disparate data types, views, or modalities into a single, unified semantic space.

Consider stereo matching in computer vision. Wenjie Luo et al. show in Efficient Deep Learning for Stereo Matching that twin branches can map left and right image patches into an embedding space where a simple dot product computes geometric disparity efficiently. This principle extends to cross-view challenges: Qiang Chen et al. leverage parallel feature extraction branches in Feature Map Pooling for Cross-View Gait Recognition Based on Silhouette Sequence Images to reconcile dramatic perspective shifts in video sequences by computing the absolute difference between aggregated feature maps.

This alignment capability is equally powerful outside of computer vision. In knowledge representation, Phillipp Müller et al. employ identical pipelines in An Integrated Graph Neural Network for Supervised Non-obvious Relationship Detection in Knowledge Graphs to ensure directionally invariant entity matching. For information retrieval, Jinsung Yoon et al. demonstrate in Search-Adaptor: Embedding Customization for Information Retrieval that applying a single set of shared weights across both query and document encoders maintains a strict consistency that outperforms asymmetric transformation layers.

Generalization in Low-Data and Unsupervised Regimes

Perhaps the most compelling property of the shared-weight dual encoder is its ability to generalize under extreme data constraints, such as few-shot or entirely unsupervised learning.

In few-shot scenarios, a network cannot learn a robust classifier from one or two examples. However, it can learn whether a new sample matches a known example. Amal Akli et al. exploit this in FlakyCat: Predicting Flaky Tests Categories using Few-Shot Learning, using twin networks to categorize software bugs based on proximity to a tiny set of exemplar instances.

When labels are completely absent, Siamese architectures enable self-supervised representation learning by comparing different "views" or augmentations of the same data point. Amirhossein Kazerouni demonstrates this in FuseNet: Self-Supervised Dual-Path Network for Medical Image Segmentation, where a shared-weight encoder processes an original image alongside its augmented variant. This forces the network to extract features invariant to noise and lighting while preserving underlying structural topology.

Taking this a step further, Guansong Pang et al. show in Unsupervised Representation Learning by Predicting Random Distances that a shared-weight architecture can learn powerful tabular features purely by training twin branches to predict distances in a randomly projected space. By framing representation learning as a relative distance-matching problem, Siamese networks bypass the need for human annotations entirely.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 534 papers
  2. Version 1 · 2026-10-04 · 555 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_siamese_network_architecture_d7fad148,
  title = {Siamese network architecture},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/siamese-network-architecture/rdr_d7fad1481a954d7185cb0fcc7cbfd0f2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_siamese_network_architecture_d7fad148_v2,
  title = {Siamese network architecture},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/siamese-network-architecture/rdr_d7fad1481a954d7185cb0fcc7cbfd0f2/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake