Implicit Non-linear Similarity Scoring for Recognizing Unseen Classes

Yuchen Guo, Guiguang Ding, Jungong Han, Sicheng Zhao, Bin Wang

Abstract

Zero-Shot Learning (ZSL) aims to recognize objects from classes not seen during training by leveraging semantic descriptions. Traditional ZSL models typically employ Explicit Linear Scoring (ELS), such as dot products, to match image features with class attributes in a joint embedding space. However, these linear methods often fail to capture the complex, non-linear relationships between high-dimensional visual data and semantic attributes. This paper introduces ICINESS (Implicit Non-linear Similarity Scoring), a framework that replaces rigid linear scoring with a learnable neural network. By fusing features through non-linear hidden layers, ICINESS approximates complex similarity functions more effectively. Our evaluation across four benchmark datasets shows that ICINESS significantly outperforms existing linear models, setting new state-of-the-art records for both ZSL and Generalized ZSL (GZSL).

Introduction

The core challenge of Zero-Shot Learning (ZSL) is bridging the gap between visual features and semantic information (e.g., attributes or word embeddings). Most existing frameworks map these two modalities into a shared "joint embedding space" and use a simple linear compatibility matrix or inner product to determine a match.

The authors argue that this reliance on Explicit Linear Scoring (ELS) is a fundamental bottleneck. The mapping between a complex image (like a bird's plumage) and a semantic attribute (like "spotted wings") is inherently non-linear. Forcing this relationship into a linear mold leads to a loss of nuanced information, limiting the model's discriminative power when encountering unseen categories.

The ICINESS Framework

To overcome the limitations of linear scoring, the authors propose ICINESS. Instead of a fixed mathematical operation, ICINESS uses a neural network to learn an implicit similarity function.

The ICINESS architecture: Image and class features are fused through hidden layers to produce a similarity score.

Key Components

  1. Feature Fusion: The model takes an image feature vector and a class attribute vector as inputs. These are concatenated and passed through one or more fully connected hidden layers.
  2. Non-linear Activation: By employing non-linear activations like ReLU or tanh, the network can approximate any continuous similarity function, as per the Universal Approximation Theorem.
  3. End-to-End Learning: The framework supports two variants:
  4. ICINESS-S (Shallow): Operates on fixed, pre-extracted features.
  5. ICINESS-D (Deep): Fine-tunes the image backbone (e.g., ResNet-101) concurrently with the similarity network.
  6. Loss Functions: The authors shift from regression-based losses to Binary Cross-Entropy or Triplet Loss, treating similarity scoring as a classification task ("match" vs. "no-match").

Experimental Evaluation

The framework was tested on four standard ZSL benchmarks: AwA2 (animals), CUB (birds), SUN (scenes), and aPY (diverse objects).

Key Insights

The experimental results provided several critical observations regarding the design of non-linear similarity scorers:

Ablation studies showing the impact of loss functions, number of layers, and hidden units across different datasets.
  • Depth and Non-linearity: Two-layer fusion networks consistently outperformed single-layer versions, confirming that the added non-linearity is essential for capturing complex correspondences.
  • Classification vs. Regression: Using classification-based losses (Cross-entropy or Ranking) proved significantly more effective than regression. Teaching the model to distinguish between matches and non-matches is more robust than predicting a specific scalar value.
  • Overfitting Risks: While fine-tuning the image backbone (Deep-Image) yielded gains, fine-tuning the class attribute model (Deep-ALL) often degraded performance. This is likely due to the small number of class vectors relative to images, leading to rapid overfitting on the semantic side.

Results

ICINESS achieved state-of-the-art results across all benchmarks. On the AwA2 dataset, ICINESS-D reached a ZSL accuracy of 67.0%, a significant jump over the 62.5% baseline. In the more challenging Generalized ZSL (GZSL) setting, the improvements were even more stark; for instance, on CUB, the H-score improved from 34.4 to 41.8. On average, the deep version of ICINESS provided a 6.48% accuracy boost over top-performing linear baselines.

Conclusion and Future Work

ICINESS demonstrates that the "linear bottleneck" in ZSL can be effectively bypassed using learnable non-linear similarity scoring. While the approach is highly effective, the authors note that training the class-side model remains unstable due to the limited number of semantic prototypes. Future research may focus on better regularization for class embeddings or integrating external knowledge graphs to further bridge the domain shift between seen and unseen classes.

Summary updated

Abstract

Recognizing unseen classes is an important task for real-world applications, due to: 1) it is common that some classes in reality have no labeled image exemplar for training; and 2) novel classes emerge rapidly. Recently, to address this task many zero-shot learning (ZSL) approaches have been proposed where explicit linear scores, like inner product score, are employed to measure the similarity between a class and an image. We argue that explicit linear scoring (ELS) seems too weak to capture complicated image-class correspondence. We propose a simple yet effective framework, called Implicit Non-linear Similarity Scoring (ICINESS). In particular, we train a scoring network which uses image and class features as input, fuses them by hidden layers, and outputs the similarity. Based on the universal approximation theorem, it can approximate the true similarity function between images and classes if a proper structure is used in an implicit non-linear way, which is more flexible and powerful. With ICINESS framework, we implement ZSL algorithms by shallow and deep networks, which yield consistently superior results.
BibTeX
@inproceedings{guo2018implicit,
  author = {Yuchen Guo and Guiguang Ding and Jungong Han and Sicheng Zhao and Bin Wang},
  title = {{Implicit Non-linear Similarity Scoring for Recognizing Unseen Classes}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/680},
}
Figure