Introduction
Linear regression models are widely utilized for their interpretability and simplicity. However, they often encounter significant challenges in "small data" scenarios, where the number of features is large relative to the number of observations. In such cases, models are prone to overfitting or producing coefficients that contradict known domain logic. While techniques like transfer learning or active learning can help, they typically require auxiliary data that may not be available.
Human experts often possess valuable "feature-level" knowledge—for instance, understanding that a specific variable should have a positive correlation with the target. Previous attempts to integrate this knowledge used "hard" constraints, which can lead to model failure if even a single piece of advice is incorrect. This paper proposes a framework for incorporating human guidance as "soft" constraints, providing a robust way to improve model performance in data-sparse environments.
Methodology
The authors introduce three regression variants that treat human guidance as soft constraints using slack variables, similar to the approach used in Support Vector Machines. This formulation allows the model to deviate from human advice if the training data provides strong evidence to the contrary.
- Parameter Sign Constrained Regression (PSCR): Constraints are placed on the expected sign (positive or negative) of specific feature coefficients.
- Parameter Relative Constrained Regression (PRCR): Constraints define the relative importance or impact of features (e.g., feature $j$ should have a larger impact than feature $k$).
- Pairwise Parameter Sign Constrained Regression (PPSCR): Constraints specify that two features should share the same or opposite signs, without necessarily knowing the absolute sign.
Additionally, the paper introduces Parameter Constraint Transfer, which automates the generation of these constraints from related source domains, and provides theoretical error bounds demonstrating the robustness of PSCR to noisy guidance.
Experimental Setup
The effectiveness of these methods was evaluated across seven diverse datasets, ranging from synthetic linear models to real-world applications in housing, healthcare, and education:
- Synthetic Linear: 10 covariates in a controlled regression task.
- Boston & King County Housing: Predicting home prices based on structural and neighborhood features.
- Wine Quality: Predicting quality from chemical characteristics.
- Concrete Strength: Predicting compressive strength based on age and composition.
- ITS (Intelligent Tutoring System): Predicting student performance in physics.
- Heart Disease: Predicting risk from lifestyle and biometric data.
Key Results and Discussion
The experimental results demonstrate that incorporating feature-level constraints consistently improves predictive accuracy over standard regularization techniques.
Accuracy and Robustness
As shown in the performance comparison, the proposed methods (particularly PSCR) frequently achieved the lowest Mean Squared Error (MSE). For example, on the ITS dataset, PSCR reached an error of 0.525, significantly outperforming Ridge (0.625) and Lasso (0.700).
A critical advantage of this framework is its robustness. In comparisons with "Signed Ridge"—which utilizes hard constraints—the soft-constraint PSCR method proved superior because it could adaptively ignore erroneous guidance. This flexibility ensures that human intuition acts as a helpful prior rather than a rigid, potentially damaging rule.
Transfer Learning via Logic
The "Parameter Constraint Transfer" method also showed significant promise. By transferring the logic of a model (signs and orderings) rather than raw predictions, it outperformed standard Transfer Ridge across all datasets. This suggests that qualitative domain knowledge is a more portable and effective inductive bias than quantitative predictions when moving between related domains.
Conclusion
This research highlights the power of "soft" feature-level constraints in bridging the gap between human expertise and machine learning. By allowing models to learn from expert intuition while maintaining the flexibility to prioritize data, the proposed framework offers a robust solution for small-data regression problems. Future work will focus on addressing the non-convexity of pairwise constraints and extending theoretical error bounds to more complex constraint types.
Summary updated