MSCH: Microbatch-Based Selective Activation Checkpointing With Recomputation Hidden for Efficient Training of LLM Models

Cheng Zhang, Minjun Yu, Li Yu, Pengyu Cong, Yuchao Yan, Jie Bao, Jian Jiang, Xiaozheng Wang, Xiaolong Ye, Tao Tang, Liang XiaoIEEE Access 2024 · DOI

BibTeX
@article{zhang2024msch,
  author = {Cheng Zhang and Minjun Yu and Li Yu and Pengyu Cong and Yuchao Yan and Jie Bao and Jian Jiang and Xiaozheng Wang and Xiaolong Ye and Tao Tang and Liang Xiao},
  title = {{MSCH: Microbatch-Based Selective Activation Checkpointing With Recomputation Hidden for Efficient Training of LLM Models}},
  journal = {IEEE Access},
  year = {2024},
  doi = {10.1109/access.2024.3456788},
}

Loading related content...

Figure