CaStRL: Context-Aware State Representation learning with Transformer

Jurat Shayiding, Mohamed Zayan, Brian D Ziebart, Xinhua ZhangOpenReview 2024 · Submission to ICLR 2024 · Withdrawn · OpenReview

Its authors withdrew this paper, so there is no full text to read or summarize.

Abstract

Learning a versatile representation from high-dimensional observation data is a crucial stepping stone for building autonomous agents capable of effective decision-making in various downstream tasks. Yet, learning such a representa- tion without additional supervisory signals poses formidable practical challenges. In this work, we introduce Context-Aware State Representation Learning (CaStRL), a novel unsupervised representation pre-training approach that combines the strength of generative autoregressive modeling with the pretraining-finetuning paradigm. To encourage CaStRL to grasp the underlying dynamics information of the environment, we enforce it to jointly learn the latent state representation along with the contexts that influence the model’s ability to learn a generalizable representation for control tasks. In CaStRL, we first employ the Video-Swin Transformer as a vision encoder, customizing it to support autoregressive modeling through the incorporation of a causal attention mask. Then, we design Context-GPT to learn context from historical sequences of state representation, which drives the model towards capturing global structural patterns by propagating information across extended time horizons. This significantly improves the adaptability of the learned representation for diverse control tasks. By emphasizing reward-free evaluation and limited data constraints in both pre-training and fine-tuning stages, we find, across a wide range of Atari experiments, that pre- trained representations can substantially facilitate downstream learning efficiency.
BibTeX
@misc{shayiding2024castrl,
  author = {Jurat Shayiding and Mohamed Zayan and Brian D Ziebart and Xinhua Zhang},
  title = {{CaStRL: Context-Aware State Representation learning with Transformer}},
  year = {2024},
  url = {https://openreview.net/forum?id=y1AJUwqsla},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2024},
}
Figure