L*ReLU: Piece-wise Linear Activation Functions for Deep Fine-grained Visual Categorization

Mina Basirat, Peter RothWACV 2020

TL;DR

This paper introduces LReLU, a novel piece-wise linear activation function designed to improve fine-grained visual categorization (FGVC) by better modeling the "degree of absence" of features. By adapting the slope of the negative part of the activation function based on the data's Lipschitz constant, LReLU achieves superior accuracy compared to standard activation functions like ReLU and even other parametric variants on challenging FGVC tasks.

What Problem Does This Paper Address?

Deep neural networks have revolutionized image categorization, but most approaches rely on a standard set of architectural choices, including the use of Rectified Linear Units (ReLUs) as activation functions. While ReLUs are computationally efficient and work well for general categorization tasks (coarse-grained visual categorization or CGVC), they struggle when subtle visual differences are crucial for distinguishing between highly similar classes. This is particularly true in Fine-Grained Visual Categorization (FGVC), where differentiating between, for example, different species of birds or models of cars, requires capturing minute details. The authors argue that existing activation functions, including ReLU and its variants like Leaky ReLU (LReLU) and Exponential Linear Units (ELU), do not adequately model the "degree of absence" of features, which is critical for FGVC. They propose that a more nuanced approach to handling negative activations is needed to improve performance on these challenging tasks.

Methods

The core contribution of this paper is the introduction of LReLU, a piece-wise linear activation function. LReLU builds upon the standard ReLU by introducing a linear component for negative inputs, similar to LReLU. However, the key innovation lies in how the slope of this negative component is determined. Instead of a fixed small value (like in LReLU) or a randomly chosen value (like in RReLU), L*ReLU sets this slope based on the data's Lipschitz constant.

The Lipschitz constant of a function measures how much its output can change relative to a change in its input. For activation functions, a smaller Lipschitz constant generally implies more stable and predictable behavior, which can lead to better generalization. The authors propose that by adapting the slope of the negative part of the activation function to match the Lipschitz properties of the data, they can effectively model the "degree of absence" of features. This allows the network to retain more information about subtle negative cues, which are vital for FGVC. The positive part of L*ReLU remains the identity function, similar to ReLU, preserving computational efficiency.

Datasets

The paper evaluates L*ReLU on a range of datasets, categorized as follows:

  • Coarse-Grained Visual Categorization (CGVC):
    • CIFAR-10
    • CIFAR-100
  • Fine-Grained Visual Categorization (FGVC):
    • Caltech-UCSD Birds-200-2011
    • Car Stanford
    • Dog Stanford
    • Aircraft
    • iFood

The authors highlight that the FGVC datasets are characterized by subtle visual differences between classes, making them significantly more challenging than CGVC datasets where classes are more distinct.

Key Insights and Ideas

  • The "Degree of Absence" Matters for FGVC: The paper strongly emphasizes that for fine-grained tasks, simply knowing a feature is absent (outputting zero) is insufficient. The network needs to understand how absent it is, which requires a more nuanced handling of negative activations.
  • Lipschitz Constant as a Data Property for AF Tuning: The core idea of using the data's Lipschitz constant to guide the selection of the activation function's slope is a novel approach. It suggests that task-specific properties can inform architectural choices beyond just network depth or layer types.
  • Task-Specific Activation Functions: The experiments reveal that different datasets (even within FGVC) benefit from different optimal slope values for L*ReLU, underscoring the need for task-specific activation functions rather than a one-size-fits-all solution.
  • L*ReLU Offers a Sweet Spot: L*ReLU strikes a balance between fixed activation functions (like ReLU) and fully parametric ones (like PReLU), offering improved performance without the complexity and potential pitfalls of learning all parameters from scratch.

Key Results

  • Superior FGVC Performance: L*ReLU consistently outperformed standard activation functions (ReLU, ELU, SELU, Swish) and parametric functions (PReLU, PSwish) across all five FGVC datasets, achieving improvements of up to 5% in accuracy.
  • CGVC Performance: For CGVC tasks (CIFAR-10 and CIFAR-100), L*ReLU also showed competitive results, particularly on CIFAR-100 where a small slope (0.1-0.25) provided a noticeable benefit over baselines.
  • Optimal Slope Range: For FGVC tasks, the authors found a stable range for the Lipschitz parameter (and thus the slope) of [0.1, 0.4], indicating that while the exact value is important, it's not overly sensitive within this range.
  • Parametric AFs are Sensitive: The comparison with PReLU and PSwish showed that their performance was highly dependent on initialization, whereas L*ReLU provided more consistent results.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the optimal slope for L*ReLU is important, it's not critical within a certain range. However, finding this optimal range still requires experimentation. The authors also note that the proposed piece-wise linear approximation might not capture all nuances of complex non-linear functions.
  • Future Work: The authors suggest several promising directions:
  • Automatically estimating the Lipschitz constant from the data.
  • Exploring the application of this Lipschitz-based approach to more complex, non-linear activation functions.
  • Investigating the theoretical underpinnings of why specific Lipschitz constants are beneficial for different FGVC tasks.

Summary updated

BibTeX
@inproceedings{basirat2020l,
  author = {Mina Basirat and PETER ROTH},
  title = {{L*ReLU: Piece-wise Linear Activation Functions for Deep Fine-grained Visual Categorization}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure