On Task Description of In-context Learning: A Study from Information Perspective

Meidai Xuanyuan, Tao Yang, Jingwen Fu, Yuwang WangOpenReview 2024 · Submission to ICLR 2024 · Rejected · OpenReview

Abstract

Large language models (LLMs) have demonstrated remarkable performance in a wide range of applications, making in-context learning an essential technique. Although the in-context learning has been widely applied, our understanding of its underlying processes still remains limited. In-context learning in LLMs primarily relies on two types of information: in-context samples and task descriptions. While previous research has extensively investigated the influence of in-context samples on learning behavior, the role of task descriptions has not been adequately explored, despite their practical significance. In this paper, we present a study examining the impact of task descriptions on the in-context learning performance of LLMs. We devise a synthetic experiment setting, making the information of task description controllable. Through a series of well-designed experiments, we systematically vary task description information and assess the resulting effects on model performance across multiple tasks. Our findings reveal complex roles of task descriptions: task description will suppress the model to learn from in-context examples; task description will increase the lower bound of the in-context learning performance. This study contributes to a deeper understanding of the in-context learning mechanism in LLMs, paving the way for more effective real-world applications of these powerful models
BibTeX
@misc{xuanyuan2024task,
  author = {Meidai Xuanyuan and Tao Yang and Jingwen Fu and Yuwang Wang},
  title = {{On Task Description of In-context Learning: A Study from Information Perspective}},
  year = {2024},
  url = {https://openreview.net/forum?id=TFR0GrzERG},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2024},
}
Figure