We introduce an on-line classification algorithm based on the hierarchical partitioning of the feature space which provides a powerful performance under the defined empirical loss. The algorithm adaptively partitions the feature space and at each region trains a different classifier. As a final classification result algorithm adaptively combines the outputs of these basic models which enables it to create a linear piecewise classifier model that can work well under highly non-linear complex data. The introduced algorithm also have scalable computational complexity that scales linearly with dimension of the feature space, depth of the partitioning and number of processed data. Through experiments we show that the introduced algorithm outperforms the state-of-the-art ensemble techniques over various well-known machine learning data sets.