A Tutorial on Regression Analysis: From Linear Models to Deep Learning - Lecture Notes on Artificial Intelligence

Jingyuan Wang, Jiahao JiarXiv 2025 · DOI · arXiv

Introduction

Regression analysis is a cornerstone of both classical statistics and modern artificial intelligence. However, it is often taught in fragments: statistics courses emphasize interpretability and Ordinary Least Squares (OLS), while machine learning curricula frequently pivot directly to the "black box" of deep neural networks. This tutorial bridges that gap, offering a unified pedagogical framework that traces the evolution of regression from simple linear models to complex deep learning architectures.

The Unified Framework of Model Construction

The core of the tutorial's methodology is a structured approach to building any regression model, likened to "the three steps for putting an elephant into a refrigerator":

  1. The Regression Function: Selecting the mathematical form of the relationship between inputs and outputs (e.g., linear, polynomial, or neural network).
  2. The Loss Function: Defining a metric to quantify the error between predictions and actual labels.
  3. Parameter Estimation: Utilizing optimization processes (like Gradient Descent) to find the best-fitting parameters.
Common regression models and their corresponding functional forms.

As illustrated in the table above, models ranging from standard Linear Regression to Autoregression (AR) share a common DNA of parameters ($\theta$) and functional forms. This continuity allows students to see how complex models are often just extensions of simpler ones.

Loss Functions and Label Types

The choice of a loss function is intrinsically tied to the nature of the data being modeled. Continuous labels typically require metrics like Mean Squared Error (MSE) or Mean Absolute Error (MAE), while discrete labels—often found in classification tasks—rely on Cross-Entropy.

Comparison of error metrics and their applicability to continuous or discrete label types.

Technical Progressions: From Linear to Deep Models

The tutorial details how linear models are extended to capture complex, non-linear patterns. By applying nonlinear transformations—such as Gaussian Radial Basis Functions (RBFs) or Fourier series—to input features, a model can remain linear in its parameters while fitting intricate data patterns.

Deep learning is presented as the ultimate evolution of this concept: adaptive regression. Instead of a human manually selecting basis functions, a neural network learns the optimal transformations through its hidden layers. This perspective demystifies backpropagation, framing it not as a magic algorithm, but as a systematic application of the chain rule to update weights based on error signals.

Key Mathematical Insights

One of the most significant takeaways is the "Unified Gradient." The tutorial demonstrates that for Linear, Logistic, and Softmax regression, the gradient of the loss function simplifies to a consistent form: (prediction - label) × input. This mathematical elegance explains why the same optimization algorithms, such as Stochastic Gradient Descent (SGD), are effective across diverse tasks.

Furthermore, the tutorial provides a geometric explanation for regularization. L1 regularization (LASSO) promotes sparsity because its "level set" is a diamond with sharp corners on the axes, making it likely for the optimal solution to zero out certain coefficients. In contrast, the circular L2 (Ridge) "level set" tends to shrink coefficients without eliminating them entirely, ensuring matrix invertibility even with correlated features.

Conclusion and Future Work

While this tutorial provides a rigorous foundation, it remains a "living document." Future expansions are expected to include a Bayesian perspective to better handle uncertainty, formal techniques for assessing feature importance, and deep dives into modern architectures like Transformers and Convolutional Neural Networks (CNNs).

Summary updated

Abstract

This article serves as the regression analysis lecture notes in the Intelligent Computing course cluster (including the courses of Artificial Intelligence, Data Mining, Machine Learning, and Pattern Recognition). It aims to provide students -- who are assumed to possess only basic university-level mathematics (i.e., with prerequisite courses in calculus, linear algebra, and probability theory) -- with a comprehensive and self-contained understanding of regression analysis without requiring any additional references. The lecture notes systematically introduce the fundamental concepts, modeling components, and theoretical foundations of regression analysis, covering linear regression, logistic regression, multinomial logistic regression, polynomial regression, basis-function models, kernel-based methods, and neural-network-based nonlinear regression. Core methodological topics include loss-function design, parameter-estimation principles, ordinary least squares, gradient-based optimization algorithms and their variants, as well as regularization techniques such as Ridge and LASSO regression. Through detailed mathematical derivations, illustrative examples, and intuitive visual explanations, the materials help students understand not only how regression models are constructed and optimized, but also how they reveal the underlying relationships between features and response variables. By bridging classical statistical modeling and modern machine-learning practice, these lecture notes aim to equip students with a solid conceptual and technical foundation for further study in advanced artificial intelligence models.
BibTeX
@misc{wang2025tutorial,
  author = {Jingyuan Wang and Jiahao Ji},
  title = {{A Tutorial on Regression Analysis: From Linear Models to Deep Learning - Lecture Notes on Artificial Intelligence}},
  year = {2025},
  eprint = {2512.04747},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.04747},
}
Figure