Joint optimization of linear and nonlinear models for sequential regression
Date
2022-12Source Title
Digital Signal Processing
Print ISSN
1051-2004
Electronic ISSN
1095-4333
Publisher
Academic Press
Volume
132
Pages
103802-1 - 103802-11
Language
English
Type
ArticleItem Usage Stats
7
views
views
9
downloads
downloads
Abstract
We investigate nonlinear regression and introduce a novel approach based on the joint optimization of linear and nonlinear models. In order to capture both the nonlinear and linear characteristics in sequential data, we model the underlying data as a combination of linear and nonlinear models, where we optimize the models jointly to minimize the final regression error. As the nonlinear model, we employ a differentiable version of the boosted decision trees. As the linear model, we use the well-known SARIMAX model. Our approach is generic so that any differentiable nonlinear or linear model can be readily employed provided that they are differentiable. By this joint optimization, we alleviate the well-known underfitting and overfitting problems in modeling sequential data. Through our experiments on synthetic and real-life data, we demonstrate significant improvements over individual components as well as the combination/mixture methods in the literature.
Keywords
Online learningRegression
Seasonal auto-regressive integrated moving average with eXogenous factors (SARIMAX)
Soft gradient boosting decision tree (Soft GBDT)
Stochastic gradient descent (SGD)