Introduction
Deep learning has traditionally relied on the availability of massive, high-quality labeled datasets. However, in specialized domains such as medicine, security, and engineering, labels are often expensive, rare, or corrupted by noise. To mitigate these challenges, researchers have increasingly turned to "informed learning"—a paradigm that integrates domain knowledge, such as physical laws (PDEs) or mathematical constraints (e.g., the Shannon capacity formula), directly into the neural network's loss function.
While informed learning has demonstrated empirical success, a rigorous mathematical understanding of its mechanisms has been lacking. Specifically, existing theory often assumes a single unique label for each data point, whereas informed learning involves multiple, potentially conflicting supervisions (human labels vs. knowledge-based constraints). This paper provides a theoretical foundation for informed learning, analyzing convergence, generalization, and sampling complexity within the regime of wide, over-parameterized networks.
Theoretical Framework: The NTK Perspective
The authors analyze Informed Deep Neural Networks (DNNs) using the Neural Tangent Kernel (NTK) framework. This approach allows for a precise characterization of the network's behavior during training when the width ($m$) is sufficiently large.
Smooth Sets and Effective Labels
A primary challenge in informed learning is the presence of multiple supervisions on nearby inputs. To address this, the authors introduce the concept of "smooth sets." They prove that a wide network does not simply choose one supervision over the other; instead, it converges to an "effective label." This effective label represents a weighted compromise between the empirical data and the domain knowledge, determined by the specific hyperparameters of the loss function.
The Generalized Informed Objective
Standard informed learning typically uses a single hyperparameter to balance data and knowledge. This paper proposes a more flexible objective utilizing two parameters, $\lambda$ and $\beta$. This decoupling is crucial for distinguishing between the two primary effects of domain knowledge: regularization and supplementation.
The Dual Role of Domain Knowledge
The research identifies that domain knowledge serves two distinct, quantifiable functions:
- Regularization: Knowledge acts as a smoothing agent. By providing a consistent constraint across the input space, it prevents the model from overfitting to noisy or corrupted labels. When labels are unreliable, the knowledge-based loss pulls the model toward a more physically or logically consistent solution.
- Supplementation: Knowledge provides a supervision signal in regions of the input space where labels are entirely missing. In this capacity, it acts as a "pseudo-labeler," effectively increasing the density of the training signal without requiring additional human intervention.
By establishing a quantitative "exchange rate," the authors show that high-quality domain knowledge can effectively replace a specific number of labeled samples, thereby reducing the overall sampling complexity required for generalization.
Key Results and Empirical Validation
Convergence and Generalization
The authors provide a formal proof of convergence (Theorem 4.1), demonstrating that wide informed DNNs are guaranteed to reach a global minimum that balances all available information. Furthermore, they derive generalization bounds showing that the population risk (the error on unseen data) is bounded by a combination of label noise, knowledge imperfectness, and the number of samples. A significant finding is the "imperfectness floor": if domain knowledge is fundamentally flawed, there is a limit to how much it can improve performance, regardless of the amount of data.
Experimental Performance
The theory was validated using both synthetic and real-world tasks:
- Bohachevsky Function: In this non-convex landscape, knowledge-based constraints helped the network navigate complex optimization surfaces.
- Wireless Link Scheduling: Using the Shannon rate formula as domain knowledge, the informed DNN achieved high accuracy with only 100 labeled samples. Without this knowledge, the model required a significantly larger dataset to reach comparable performance.
- Noise Robustness: Experiments demonstrated that when labels were noisy ($\sigma_z^2 = 0.1$), a properly tuned informed DNN could achieve error rates nearly identical to a model trained on perfect labels.
Limitations and Future Directions
While this work provides a landmark theoretical basis for informed learning, it relies on the assumption of extreme over-parameterization (very wide networks) to satisfy the NTK conditions. In practice, smaller "narrow" networks are often used in edge computing, and their convergence properties in the context of informed learning remain an open question. Additionally, the current analysis assumes the network's internal representations remain relatively static (the "lazy training" regime) and does not fully account for active feature learning. Future work may also extend these proofs to non-convex loss landscapes common in complex physics-informed models.
Summary updated