Ten Key Problems of Pre-trained Models: An Outlook of Representation Learning

Ning Ding, Weize Chen, Zhengyan Zhang, Shengding Hu, Ganqu Cui, Yuan Yao, Yujia Qin, Zheni Zeng, Xu Han, Zhiyuan Liu, Yankai Lin, Maosong SunOpenalex Unknown 2023 · DOI · OpenAlex

Abstract

Large-scale pre-trained models (PTMs) have redefined the state-of-the-art in representation learning, particularly within Natural Language Processing (NLP). As we enter an era dominated by these "foundation models," the field faces a critical transition from empirical scaling to addressing fundamental architectural, theoretical, and systemic challenges. This paper outlines ten key open problems that will shape the next decade of AI research, focusing on efficiency, cognitive integration, and the establishment of rigorous theoretical foundations.

Introduction: Standing on the Shoulders of Giants

The rapid evolution of models like GPT and BERT has transformed AI from a collection of specialized tasks into a unified field of "big models." However, our current progress relies heavily on massive compute and empirical observation rather than deep understanding. We are currently standing on the shoulders of giants without a clear map of their internal mechanics or their ultimate destination. To move forward, the research community must bridge the gap between scaling success and the lack of theoretical clarity, efficiency, and controllable safety.

A Strategic Roadmap: Ten Key Problems

The future of representation learning can be categorized into several critical domains:

1. Architectural Evolution

The Transformer architecture, while dominant, may not be the final word in AI. Future research is pivoting toward:

  • Dynamical Systems: Treating neural network layers as time steps in Ordinary Differential Equations (ODEs) to improve stability and depth.
  • Non-Euclidean Geometry: Adapting models to handle complex graph structures and non-linear data manifolds.
  • Neuroscience-Inspired Designs: Leveraging Spiking Neural Networks (SNNs) and Mixture of Experts (MoE) to achieve the energy efficiency and spatial sparsity of the human brain.

2. Efficiency and Adaptation

As models grow, full-parameter fine-tuning becomes prohibitively expensive. The rise of "Delta Tuning" (e.g., LoRA, prefix-tuning) allows researchers to adapt massive models by modifying less than 1% of their parameters. This shift democratizes AI research and enables more sustainable development.

3. Cognitive Integration

True intelligence requires more than token prediction. The next generation of models must integrate:

  • Tool Use: Interacting with external databases, search engines, and APIs.
  • Complex Reasoning: Enhancing "Chain-of-Thought" capabilities to ensure logical consistency.
  • Real-World Grounding: Moving beyond text to "omni-perception" frameworks that unify text, vision, and audio.

4. Systemic Infrastructure

Analogous to how Database Management Systems (DBMS) revolutionized data handling, we need Big Model Systems (BMS). A BMS would provide a unified management layer, abstracting the complexities of deep learning programming and making large-scale models accessible to a broader range of developers.

Key Insights: Emergence and Unification

A central mystery in modern AI is "Emergence"—the phenomenon where specific capabilities, such as in-context learning, appear suddenly once a model reaches a certain scale. Understanding the mathematical "trigger points" for these capabilities is a primary goal. Simultaneously, we are witnessing a "Grand Unified Theory" of architectures, where the Transformer backbone is successfully applied across NLP, Computer Vision, and even biological tasks like protein folding.

Limitations and Ethical Frontiers

Despite their success, PTMs face significant hurdles:

  • Theoretical Void: We lack a complete mathematical framework to explain why these models work as well as they do.
  • Safety and Controllability: Identifying and disabling harmful behaviors within a "black box" of billions of parameters remains an unsolved technical challenge.
  • Evaluation Gaps: Current metrics (like BERT-Score) are insufficient for evaluating controllable generation or the ethical alignment of models across diverse social contexts.

Conclusion

The next era of representation learning will be defined by a shift from "bigger is better" to "smarter and more efficient." By addressing these ten key problems—ranging from bio-inspired architectures to the creation of robust model management systems—the AI community can move toward a more grounded, theoretical, and safe intelligence.

Summary updated

Abstract

Abstract The aforementioned representation learning methods have shown their effectiveness in various NLP scenarios and tasks. Large-scale pre-trained language models (i.e., big models) are the state of the art of representation learning for NLP and beyond. With the rapid growth of data scale and the development of computation devices, big models bring us to a new era of AI and NLP. Standing on the new giants of big models, there are many new challenges and opportunities for representation learning. In the last chapter, we will provide a 2023 outlook for the future directions of representation learning techniques for NLP by summarizing ten key open problems for pre-trained models.
BibTeX
@inproceedings{ding2023ten,
  author = {Ning Ding and Weize Chen and Zhengyan Zhang and Shengding Hu and Ganqu Cui and Yuan Yao and Yujia Qin and Zheni Zeng and Xu Han and Zhiyuan Liu and Yankai Lin and Maosong Sun},
  title = {{Ten Key Problems of Pre-trained Models: An Outlook of Representation Learning}},
  booktitle = {Openalex Unknown},
  year = {2023},
  doi = {10.1007/978-981-99-1600-9\_14},
}
Figure