In this paper, we study the binary classification problem in machine learning and introduce a novel classification algorithm based on the 'Context Tree Weighting Method'. The introduced algorithm incrementally learns a classification model through sequential updates in the course of a given data stream, i.e., each data point is processed only once and forgotten after the classifier is updated, and asymptotically achieves the performance of the best piecewise linear classifiers defined by the 'context tree'. Since the computational complexity is only linear in the depth of the context tree, our algorithm is highly scalable and appropriate for real time processing. We present experimental results on several benchmark data sets and demonstrate that our method provides significant computational improvement both in the test (5 ∼ 35×) and training phases (40 ∼ 1000×), while achieving high classification accuracy in comparison to the SVM with RBF kernel. © 2013 IEEE.