Joint optimization of linear and nonlinear models for sequential regression

Date
2022-12
Advisor
Supervisor
Co-Advisor
Co-Supervisor
Instructor
Source Title
Digital Signal Processing
Print ISSN
1051-2004
Electronic ISSN
1095-4333
Publisher
Academic Press
Volume
132
Issue
Pages
103802-1 - 103802-11
Language
English
Type
Article
Journal Title
Journal ISSN
Volume Title
Series
Abstract

We investigate nonlinear regression and introduce a novel approach based on the joint optimization of linear and nonlinear models. In order to capture both the nonlinear and linear characteristics in sequential data, we model the underlying data as a combination of linear and nonlinear models, where we optimize the models jointly to minimize the final regression error. As the nonlinear model, we employ a differentiable version of the boosted decision trees. As the linear model, we use the well-known SARIMAX model. Our approach is generic so that any differentiable nonlinear or linear model can be readily employed provided that they are differentiable. By this joint optimization, we alleviate the well-known underfitting and overfitting problems in modeling sequential data. Through our experiments on synthetic and real-life data, we demonstrate significant improvements over individual components as well as the combination/mixture methods in the literature.

Course
Other identifiers
Book Title
Keywords
Online learning, Regression, Seasonal auto-regressive integrated moving average with eXogenous factors (SARIMAX), Soft gradient boosting decision tree (Soft GBDT), Stochastic gradient descent (SGD)
Citation
Published Version (Please cite this version)