Abstract
Zero-Shot Learning (ZSL) aims to recognize objects from classes not seen during training by leveraging semantic descriptions. Traditional ZSL models typically employ Explicit Linear Scoring (ELS), such as dot products, to match image features with class attributes in a joint embedding space. However, these linear methods often fail to capture the complex, non-linear relationships between high-dimensional visual data and semantic attributes. This paper introduces ICINESS (Implicit Non-linear Similarity Scoring), a framework that replaces rigid linear scoring with a learnable neural network. By fusing features through non-linear hidden layers, ICINESS approximates complex similarity functions more effectively. Our evaluation across four benchmark datasets shows that ICINESS significantly outperforms existing linear models, setting new state-of-the-art records for both ZSL and Generalized ZSL (GZSL).
Introduction
The core challenge of Zero-Shot Learning (ZSL) is bridging the gap between visual features and semantic information (e.g., attributes or word embeddings). Most existing frameworks map these two modalities into a shared "joint embedding space" and use a simple linear compatibility matrix or inner product to determine a match.
The authors argue that this reliance on Explicit Linear Scoring (ELS) is a fundamental bottleneck. The mapping between a complex image (like a bird's plumage) and a semantic attribute (like "spotted wings") is inherently non-linear. Forcing this relationship into a linear mold leads to a loss of nuanced information, limiting the model's discriminative power when encountering unseen categories.
The ICINESS Framework
To overcome the limitations of linear scoring, the authors propose ICINESS. Instead of a fixed mathematical operation, ICINESS uses a neural network to learn an implicit similarity function.
Key Components
- Feature Fusion: The model takes an image feature vector and a class attribute vector as inputs. These are concatenated and passed through one or more fully connected hidden layers.
- Non-linear Activation: By employing non-linear activations like
ReLUortanh, the network can approximate any continuous similarity function, as per the Universal Approximation Theorem. - End-to-End Learning: The framework supports two variants:
- ICINESS-S (Shallow): Operates on fixed, pre-extracted features.
- ICINESS-D (Deep): Fine-tunes the image backbone (e.g., ResNet-101) concurrently with the similarity network.
- Loss Functions: The authors shift from regression-based losses to Binary Cross-Entropy or Triplet Loss, treating similarity scoring as a classification task ("match" vs. "no-match").
Experimental Evaluation
The framework was tested on four standard ZSL benchmarks: AwA2 (animals), CUB (birds), SUN (scenes), and aPY (diverse objects).
Key Insights
The experimental results provided several critical observations regarding the design of non-linear similarity scorers:
- Depth and Non-linearity: Two-layer fusion networks consistently outperformed single-layer versions, confirming that the added non-linearity is essential for capturing complex correspondences.
- Classification vs. Regression: Using classification-based losses (Cross-entropy or Ranking) proved significantly more effective than regression. Teaching the model to distinguish between matches and non-matches is more robust than predicting a specific scalar value.
- Overfitting Risks: While fine-tuning the image backbone (Deep-Image) yielded gains, fine-tuning the class attribute model (Deep-ALL) often degraded performance. This is likely due to the small number of class vectors relative to images, leading to rapid overfitting on the semantic side.
Results
ICINESS achieved state-of-the-art results across all benchmarks. On the AwA2 dataset, ICINESS-D reached a ZSL accuracy of 67.0%, a significant jump over the 62.5% baseline. In the more challenging Generalized ZSL (GZSL) setting, the improvements were even more stark; for instance, on CUB, the H-score improved from 34.4 to 41.8. On average, the deep version of ICINESS provided a 6.48% accuracy boost over top-performing linear baselines.
Conclusion and Future Work
ICINESS demonstrates that the "linear bottleneck" in ZSL can be effectively bypassed using learnable non-linear similarity scoring. While the approach is highly effective, the authors note that training the class-side model remains unstable due to the limited number of semantic prototypes. Future research may focus on better regularization for class embeddings or integrating external knowledge graphs to further bridge the domain shift between seen and unseen classes.
Summary updated