Representation Learning via Adversarially-Contrastive Optimal Transport

Anoop Cherian, Shuchin Aeron

Abstract

In this paper, we study the problem of learning compact (low-dimensional) representations for sequential data that captures its implicit spatio-temporal cues. To maximize extraction of such informative cues from the data, we set the problem within the context of contrastive representation learning and to that end propose a novel objective via optimal transport. Specifically, our formulation seeks a low-dimensional subspace representation of the data that jointly (i) maximizes the distance of the data (embedded in this subspace) from an adversarial data distribution under the optimal transport, a.k.a. the Wasserstein distance, (ii) captures the temporal order, and (iii) minimizes the data distortion. To generate the adversarial distribution, we propose a novel framework connecting Wasserstein GANs with a classifier, allowing a principled mechanism for producing good negative distributions for contrastive learning, which is currently a challenging problem. Our full objective is cast as a subspace learning problem on the Grassmann manifold and solved via Riemannian optimization. To empirically study our formulation, we provide experiments on the task of human action recognition in video sequences. Our results demonstrate competitive performance against challenging baselines.

Introduction

Learning effective representations for sequential data, particularly video, is complicated by variable sequence lengths and the prevalence of nuisance factors like background clutter. Contrastive learning offers a solution by training models to distinguish between positive and negative examples. However, the performance of these models is highly dependent on the "hardness" of the negative samples. If negatives are easily distinguishable (e.g., random noise), the model fails to learn the subtle spatio-temporal features necessary for complex tasks like action recognition. This paper introduces Adversarially-Contrastive Optimal Transport (ACOT), a framework designed to generate and leverage distributionally similar "hard" negatives to learn robust, low-dimensional representations.

Methodology

The ACOT framework is built upon three core pillars:

  1. Contrastive Optimal Transport (COT): Rather than simple point-to-point comparisons, ACOT uses the Wasserstein distance to compare the distribution of video features against a distribution of negative samples. This provides a global measure of contrast that is more principled than traditional contrastive losses.
  2. Adversarial Noise Generation: To create challenging negatives, the authors employ a Wasserstein GAN (WGAN) conditioned on the input data. This generator is trained to produce noise that, when added to the features, causes a pre-trained classifier to fail. This forces the learned representation to focus on the specific discriminative features the adversary attempts to corrupt.
  3. Grassmann Subspace Learning: Video sequences are represented as points on a Grassmann manifold—specifically, as low-dimensional linear subspaces. This geometric approach naturally accommodates variable-length sequences and allows for efficient Riemannian optimization. The objective function also incorporates temporal constraints to preserve the chronological flow of the data.
Overview of the ACOT framework, illustrating the flow from input features to adversarial noise generation and subspace learning.

Experiments and Results

The authors evaluated ACOT on two standard human action recognition benchmarks: JHMDB and HMDB, using both VGG-16 (2D) and I3D (3D) pre-trained features.

Performance Benchmarks

ACOT demonstrated state-of-the-art performance on the JHMDB dataset using I3D features, achieving 87.5% accuracy, which surpasses strong baselines like "I3D + Potion" (85.5%). On the larger and more challenging HMDB dataset, the full ACOT model reached 81.8% accuracy, showing a clear improvement over standard I3D pooling (80.9%).

Ablation Studies and Hyperparameter Sensitivity

A key finding of the study is the superiority of adversarial noise over random noise. On the JHMDB dataset with VGG features, ACOT reached 79.8% accuracy, while a version using random noise (COT + Random) only achieved 77.9%.

Ablation studies showing the impact of various hyperparameters on model accuracy across different feature types.

As shown in the ablation plots, model performance is influenced by several factors:

  • Number of Subspaces ($k$): Accuracy generally improves as the number of subspaces increases, allowing for a richer representation.
  • Regularization ($\beta_1, \beta_2$): Proper tuning of distortion and temporal regularization is essential for balancing data fidelity with temporal coherence.
  • Number of Negatives: Performance peaks at a moderate number of negatives (around 100), after which the optimization problem likely becomes too complex, leading to a decline in accuracy.

Conclusion

ACOT presents a mathematically elegant and empirically effective framework for video representation learning. By combining optimal transport with adversarial negative generation and manifold optimization, it successfully captures critical spatio-temporal cues while remaining robust to nuisance factors. While the computational cost of OT distance calculations remains a challenge, the method's performance on standard benchmarks highlights its potential. Future research will explore end-to-end integration of the ACOT loss into feature extraction networks.

Summary updated

Abstract

In this paper, we study the problem of learning compact (low-dimensional) representations for sequential data that captures its implicit spatio-temporal cues. To maximize extraction of such informative cues from the data, we set the problem within the context of contrastive representation learning and to that end propose a novel objective via optimal transport. Specifically, our formulation seeks a low-dimensional subspace representation of the data that jointly (i) maximizes the distance of the data (embedded in this subspace) from an adversarial data distribution under the optimal transport, a.k.a. the Wasserstein distance, (ii) captures the temporal order, and (iii) minimizes the data distortion. To generate the adversarial distribution, we propose a novel framework connecting Wasserstein GANs with a classifier, allowing a principled mechanism for producing good negative distributions for contrastive learning, which is currently a challenging problem. Our full objective is cast as a subspace learning problem on the Grassmann manifold and solved via Riemannian optimization. To empirically study our formulation, we provide experiments on the task of human action recognition in video sequences. Our results demonstrate competitive performance against challenging baselines.
BibTeX
@inproceedings{cherian2020representation,
  author = {Anoop Cherian and Shuchin Aeron},
  title = {{Representation Learning via Adversarially-Contrastive Optimal Transport}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=-iYiLhunBj},
}
Figure