Extending the Boosting Framework based on Bayesian Methodology
Doctoral thesis
Date of Examination:2023-01-24
Date of issue:2023-03-31
Advisor:Prof. Dr. Elisabeth Bergherr
Referee:Prof. Dr. Elisabeth Bergherr
Referee:Prof. Dr. Thomas Kneib
Referee:Dr. Tobias Hepp
Files in this item
Name:Zhang_dissertation.pdf
Size:2.74Mb
Format:PDF
Abstract
English
The boosting technique emerged from machine learning has become a widely used method to estimate statistical models. As one of the most successful variants, componentwise gradient boosting has been favored by more and more statisticians since its iterative procedure not only provides intuitive variable selection in high-dimensional analysis, but also supplies additional flexibility to estimate various types of additive regression terms. But its dogmatic estimates, i.e., its direct and unquestionable estimation conclusion, do not deliver any information about the error risk of estimation and prediction, which, however, is the basis for many statistical analyses. As one of the most essential conventional statistical theories, Bayesian methodology maintains the ability to quantify uncertainty. Due to its unique prior philosophy, it has grown immensely in the past decades and has led to the development of innumerable new models. However, it often fails to give precise and unambiguous guidelines for the variable selection, which in turn is the advantage of boosting. This thesis proposes a Bayesian-based boosting theory, which integrates Bayesian inference in the boosting framework. Componentwise boosting guarantees the high-dimensional analysis and the flexibility of base-learners since additive terms are updated individually. Furthermore, each base-learner inferred by Bayesian inference also preserves additional Bayesian properties such as the prior and the credible-based uncertainty quantification. The proposed Bayesian-based boosting method combines the strengths of the two approaches and overcomes the weaknesses of both. This thesis firstly solves the problem of imbalanced updates of predictors in generalized additive models for location, scale and shape (GAMLSS) estimated using gradient boosting by introducing the adaptive step-length. Then, through the implementation of Bayesian learners in the gradient boosting framework for linear mixed models (LMM), the validity of the combination of Bayesian and boosting concepts is preliminarily verified. The complete Bayesian-based boosting framework is eventually presented by applying it to a generalized model family, namely structured additive regression (STAR) models. Overall, the proposed Bayesian-based boosting is not only the first systematic study of the fusion of Bayesian inference and boosting techniques, but also an attempt to integrate machine learning and statistics at a deeper level.
Keywords: Boosting; Bayesian methodology; High-dimension; Uncertainty; Variable selection; Step-length