Per-Sample Kernel Adaptation for Visual Recognition and Grouping

Borislav Antic, Björn OmmerICCV 2015 · OpenReview

Abstract

Object, action, or scene representations that are corrupted by noise significantly impair the performance of visual recognition. Typically, partial occlusion, clutter, or excessive articulation affects only a subset of all feature dimensions and, most importantly, different dimensions are corrupted in different samples. Nevertheless, the common approach to this problem in feature selection and kernel methods is to down-weight or eliminate entire training samples or the same dimensions of all samples. Thus, valuable signal is lost, resulting in suboptimal classification. Our goal is, therefore, to adjust the contribution of individual feature dimensions when comparing any two samples and computing their similarity. Consequently, per-sample selection of informative dimensions is directly integrated into kernel computation. The interrelated problems of learning the parameters of a kernel classifier and determining the informative components of each sample are then addressed in a joint objective function. The approach can be integrated into the learning stage of any kernel-based visual recognition problem and it does not affect the computational performance in the retrieval phase. Experiments on diverse challenges of action recognition in videos and indoor scene classification show the general applicability of the approach and its ability to improve learning of visual representations.

The Problem: Stochastic and Local Noise

In visual recognition tasks, such as identifying actions in videos or classifying indoor scenes, data is rarely pristine. A person performing a "long jump" might be partially obscured by a fence, or a "kitchen" scene might contain a distracting glare on a specific counter.

Traditional methods for handling this noise are often too broad:

  1. Global Feature Selection: Deletes a specific feature (e.g., color) for every image in the dataset, even if it is only noisy in a few.
  2. Sample Weighting (SVM): Down-weights an entire image if it is noisy, potentially discarding the 90% of its features that are perfectly informative.

Noise in visual data is stochastic and local; occlusion affects different dimensions in different samples. This research proposes a "scalpel" approach: Per-Sample Kernel Adaptation (PSKA), which identifies and ignores specific corrupted feature dimensions on a per-image basis.

Methodology: Per-Sample Kernel Adaptation

The core innovation is the introduction of a reliability vector ($z_i$) for every sample $i$ in the training set.

Conceptual overview of Per-Sample Kernel Adaptation (PSKA) versus traditional feature and sample selection.

The Adapted Kernel Trick

In standard kernel methods (like SVMs with an RBF kernel), similarity is calculated by summing distances across all feature dimensions. PSKA modifies this so that the distance in dimension $l$ between samples $i$ and $j$ is weighted by $z_{il} \cdot z_{jl}$. If either sample has a low reliability score for that dimension, its contribution to the final similarity is suppressed.

Joint Optimization and Sparsity

The model learns the SVM classifier parameters and the reliability vectors simultaneously through a joint objective function. To prevent the model from discarding too much data, an $\ell_1$-regularization (sparsity prior) is applied, assuming that most data is generally useful. Because the resulting optimization is non-convex, the authors employ the Concave-Convex Procedure (CCCP) to break the problem into manageable, convex sub-problems.

Experiments and Key Results

The authors evaluated PSKA on two primary benchmarks: the Olympic Sports Dataset (783 action videos) and the MIT Indoor Dataset (6,700 scene images).

  • Action Recognition: On the Olympic Sports dataset, PSKA achieved 89.4% Mean Average Precision (mAP), rising to 91.6% with post-processing. This significantly outperformed standard feature selection methods.
  • Scene Classification: The method achieved a 17.7% improvement in mAP over the baseline (RM2C) on the MIT Indoor dataset.
  • Robustness: The model excelled in categories with high variability. For example, in the "triple jump" category, performance improved by over 30% compared to the baseline.
Impact of average sparsity on Mean Average Precision (mAP), demonstrating the effectiveness of the sparsity prior.

Insights and Future Work

The fundamental insight of PSKA is that "informativeness" is not a fixed property of a feature type, but a property of that feature within a specific context. By suppressing only noisy dimensions, the model retains the valuable signal in the rest of the feature vector. Furthermore, this refined similarity measure allows for Action Reconstruction, where the model can filter out background movement to reconstruct a "clean" version of a noisy video.

Future work may address the current assumption that all features in a test query are reliable, potentially developing methods to estimate test-sample reliability on the fly. Additionally, extending the derivation from Generalized Radial Basis Functions (GRBF) to other non-additive kernels remains a promising direction.

Summary updated

Abstract

Object, action, or scene representations that are corrupted by noise significantly impair the performance of visual recognition. Typically, partial occlusion, clutter, or excessive articulation affects only a subset of all feature dimensions and, most importantly, different dimensions are corrupted in different samples. Nevertheless, the common approach to this problem in feature selection and kernel methods is to down-weight or eliminate entire training samples or the same dimensions of all samples. Thus, valuable signal is lost, resulting in suboptimal classification. Our goal is, therefore, to adjust the contribution of individual feature dimensions when comparing any two samples and computing their similarity. Consequently, per-sample selection of informative dimensions is directly integrated into kernel computation. The interrelated problems of learning the parameters of a kernel classifier and determining the informative components of each sample are then addressed in a joint objective function. The approach can be integrated into the learning stage of any kernel-based visual recognition problem and it does not affect the computational performance in the retrieval phase. Experiments on diverse challenges of action recognition in videos and indoor scene classification show the general applicability of the approach and its ability to improve learning of visual representations.
BibTeX
@inproceedings{antic2015per,
  author = {Borislav Antic and Björn Ommer},
  title = {{Per-Sample Kernel Adaptation for Visual Recognition and Grouping}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  url = {https://openreview.net/forum?id=ryb1tgzOWr},
}
Figure