Fine-Tuning Multimodal Transformer Models for Generating Actions in Virtual and Real Environments

Aleksei Staroverov, Andrey S. Gorodetsky, Andrei S. Krishtopik, Uliana A. Izmesteva, Dmitry A. Yudin, Alexey K. Kovalev, Aleksandr I. PanovIEEE Access 2023 · DOI · OpenReview

Abstract

In this work, we propose and investigate an original approach to using a pre-trained multimodal transformer of a specialized architecture for controlling a robotic agent in an object manipulation task based on language instruction, which we refer to as RozumFormer. Our model is based on a bimodal (text-image) transformer architecture originally trained for solving tasks that use one or both modalities, such as language modeling, visual question answering, image captioning, text recognition, text-to-image generation, etc. The discussed model has been adapted for robotic manipulation tasks by organizing the input sequence of tokens in a particular way, consisting of tokens for text, images, and actions. We have demonstrated that such a model adapts well to new tasks and shows better results with fine-tuning than complete training in simulation and real environments. To transfer the model from the simulator to a real robot, new datasets were collected and annotated. In addition, experiments controlling the agent in a visual environment using reinforcement learning have shown that fine-tuning the model with a mixed dataset that includes examples from the initial visual-linguistic tasks only slightly decreases performance on these tasks, simplifying the addition of tasks from another domain.
BibTeX
@article{staroverov2023fine,
  author = {Aleksei Staroverov and Andrey S. Gorodetsky and Andrei S. Krishtopik and Uliana A. Izmesteva and Dmitry A. Yudin and Alexey K. Kovalev and Aleksandr I. Panov},
  title = {{Fine-Tuning Multimodal Transformer Models for Generating Actions in Virtual and Real Environments}},
  journal = {IEEE Access},
  year = {2023},
  doi = {10.1109/access.2023.3334791},
}

Loading related content...

Figure