Abstract
Advocacy Learning is a novel competitive training framework where specialized subnetworks, termed "Advocates," compete to convince a central "Judge" that an input belongs to their specific class. Unlike traditional cooperative neural networks, this adversarial setup forces Advocates to generate class-conditional representations—"arguments"—that highlight discriminative features. Experimental results on MNIST, Fashion-MNIST, and CIFAR-10 demonstrate that this competitive pressure leads to more robust feature extraction, improved classification accuracy, and enhanced interpretability compared to standard attention-based models.
Introduction
Most modern neural networks are designed to be cooperative: every layer and subnetwork works toward a single, shared goal. However, in many human systems, such as legal proceedings or economic markets, competition serves as a powerful mechanism for uncovering truth and refining evidence. In the context of computer vision, standard attention mechanisms typically produce a single, class-agnostic saliency map. This research explores whether introducing multiple, class-specific attention maps that compete against one another can lead to superior feature extraction and more interpretable representations.
Methods
The authors propose the Advocacy Net, a framework composed of two primary elements:
- N Advocates: Each Advocate is a subnetwork specialized for one of the $N$ possible classes. Given an input $x$, an Advocate produces an "argument" $a^i$—an attention mask that highlights features it believes support its assigned class.
- The Judge: A classification network $J$ that receives the "evidence" $E$ (the element-wise product of the input and the Advocates' masks) and produces a final prediction $\hat{y}$.
The training process involves a unique "deceptive" objective. Advocates are rewarded for maximizing the Judge's confidence in their specific class, regardless of the input's ground truth. This forces Advocates to find the most persuasive features possible. Simultaneously, the Judge is trained to correctly identify the true class, learning to distinguish valid evidence from deceptive "noise" or "bad arguments."
Datasets and Evaluation
The approach was evaluated across several benchmarks:
- MNIST & Fashion-MNIST: Standard digit and clothing classification.
- CIFAR-10: Complex object recognition.
- MIMIC-III: Electronic health records for mortality prediction (time-series data).
- Modified MNIST: Variants testing class imbalance and binary classification.
Key Insights
- Deception as a Feature: Surprisingly, the "deceptive" Advocacy Net often outperformed the "Honest" version (where Advocates only update on correct classes). The pressure to fool the Judge forces Advocates to identify highly discriminative features, which in turn improves the Judge's ability to detect subtle class differences.
- Class-Conditional Focus: Unlike standard class-agnostic attention, Advocacy Learning produces class-conditional maps. For instance, when processing a "shirt" image, a "pants" Advocate might look for vertical leg-like structures, while the "shirt" Advocate focuses on the collar and sleeves.
- Morphological Awareness: The model excels at distinguishing classes with similar global shapes (e.g., shirts vs. coats) by highlighting the specific morphological differences required to "win" the classification argument.
Key Results
- Accuracy Gains: On Fashion-MNIST, the Advocacy Net achieved 91.62% accuracy, outperforming the standard Attention Net baseline of 89.71%.
- Error Reduction: The model demonstrated a 31% reduction in error rate on MNIST and a 19% reduction on Fashion-MNIST compared to single-attention baselines.
- Robustness to Imbalance: Advocacy learning proved more robust in imbalanced data scenarios, maintaining higher accuracy for minority classes.
- Domain Specificity: The method was less effective on the MIMIC-III dataset, suggesting that spatial attention masks do not translate directly to the feature vectors used in clinical time-series data.
Limitations and Future Work
While promising, the current architecture requires one Advocate per class, which may limit scalability for datasets with thousands of classes (e.g., ImageNet). Performance is also sensitive to the relative capacity of the Judge and Advocates; an overly powerful Judge can ignore Advocates, while overly powerful Advocates can overwhelm the Judge with noise. Future work includes exploring class hierarchies to reduce the number of Advocates and further investigating the interpretability of the Judge's decision-making process.
Summary updated