We introduce the boosting notion of machine learning to the adaptive signal processing literature. In our framework, we have several adaptive filtering algorithms, i.e., the weak learners, that run in parallel on a common task such as equalization, classification, regression or filtering. We specifically provide theoretical bounds for the performance improvement of our proposed algorithms over the conventional adaptive filtering methods under some widely used statistical assumptions. We demonstrate an intrinsic relationship, in terms of boosting, between the adaptive mixture-of-experts and data reuse algorithms. Additionally, we introduce a boosting algorithm based on random updates that is significantly faster than the conventional boosting methods and other variants of our proposed algorithms while achieving an enhanced performance gain. Hence, the random updates method is specifically applicable to the fast and high dimensional streaming data. Specifically, we investigate Recursive Least Square-based and Least Mean Square-based linear and piecewise-linear regression algorithms in a mixture-of-experts setting and provide several variants of these well-known adaptation methods. Furthermore, we provide theoretical bounds for the computational complexity of our proposed algorithms. We demonstrate substantial performance gains in terms of mean squared error over the base learners through an extensive set of benchmark real data sets and simulated examples.