Novel joint optimization of gradient boosting decision trees and SARIMAX models for nonlinear time series regression using a state space approach

Abstract

We investigate nonlinear regression or forecasting in an online setting and propose an end-to-end ensemble model that addresses two key challenges: the trade-off between linear and nonlinear modeling, and the disjoint nature of the optimiza tion process. The proposed architecture seamlessly integrates a linear time series model and a nonlinear soft decision tree-based model within a unified struc ture. Specifically, the architecture employs a Seasonal Autoregressive Integrated Moving Average with Exogenous Regressors (SARIMAX) model to capture key linear patterns in time series data, such as seasonality and trend, alongside a soft gradient boosting decision tree (SGBDT), which is adept at modeling nonlinear dependencies and extracting features directly from raw inputs. Different from existing state-of-the-art hybrid approaches that typically rely on sequential and disjoint training strategies, for the first time in the literature, we introduce a jointly optimized hybrid model in which both the linear and nonlinear compo nents are trained simultaneously. This is achieved through a novel formulation based on state-space representations, which allows the integration of the ARMA based SARIMAX model and the SGBDT into a single state-space framework. We employ the Extended Kalman Filter (EKF), a nonlinear optimization tech nique, to efficiently perform the joint training process by leveraging derived state transition and measurement equations. Furthermore, the architecture is modular and extensible, allowing for the substitution or addition of alternative nonlinear models and other linear time series techniques, such as Exponential Smoothing (ETS), as long as state representations are obtained. The optimization compo nent is also flexible, enabling the replacement of the EKF with other techniques such as the Unscented Kalman Filter (UKF) or Particle Filters (PF). Through extensive experiments on well-known real-world datasets, the proposed approach demonstrates superior performance. We provide the source code as a publicly available repository to support further research and ensure reproducibility.