Feature selection using stochastic approximation with Barzilai and Borwein non-monotone gains

Simple item page
buir.contributor.authorMalekipirbazari, Milad
buir.contributor.orcidMalekipirbazari, Milad|0000-0002-3212-6498
dc.citation.epage105334-14en_US
dc.citation.spage105334-1en_US
dc.citation.volumeNumber132en_US
dc.contributor.authorAksakallı, V.
dc.contributor.authorYenice, Z. D.
dc.contributor.authorMalekipirbazari, Milad
dc.contributor.authorKargar, Kamyar
dc.date.accessioned2022-02-02T10:23:39Z
dc.date.available2022-02-02T10:23:39Z
dc.date.issued2021-08
dc.departmentDepartment of Industrial Engineeringen_US
dc.description.abstractWith recent emergence of machine learning problems with massive number of features, feature selection (FS) has become an ever-increasingly important tool to mitigate the effects of the so-called curse of dimensionality. FS aims to eliminate redundant and irrelevant features for models that are faster to train, easier to understand, and less prone to overfitting. This study presents a wrapper FS method based on Simultaneous Perturbation Stochastic Approximation (SPSA) with Barzilai and Borwein (BB) non-monotone gains within a pseudo-gradient descent framework wherein performance is measured via cross-validation. We illustrate that SPSA with BB gains (SPSA-BB) provides dramatic improvements in terms of the number of iterations for convergence with minimal degradation in cross-validated error performance over the current state-of-the art approach with monotone gains (SPSA-MON). In addition, SPSA-BB requires only one internal parameter and therefore it eliminates the need for careful fine-tuning of numerous other internal parameters as in SPSA-MON or comparable meta-heuristic FS methods such as genetic algorithms (GA). Our particular implementation includes gradient averaging as well as gain smoothing for better convergence properties. We present computational experiments on various public datasets with Nearest Neighbors and Naive Bayes classifiers as wrappers. We present comparisons of SPSA-BB against full set of features, SPSA-MON, as well as seven popular meta-heuristics based FS algorithms including GA and particle swarm optimization. Our results indicate that SPSA-BB converges to a good feature set in about 50 iterations on the average regardless of the number of features (whether a dozen or more than 1000 features) and its performance is quite competitive. SPSA-BB can be considered extremely fast for a wrapper method and therefore it stands as a high-performing new feature selection method that is also computationally feasible in practice.en_US
dc.embargo.release2024-08-31
dc.identifier.doi10.1016/j.cor.2021.105334en_US
dc.identifier.issn0305-0548
dc.identifier.urihttp://hdl.handle.net/11693/76963
dc.language.isoEnglishen_US
dc.publisherElsevier Ltden_US
dc.relation.isversionofhttps://doi.org/10.1016/j.cor.2021.105334en_US
dc.source.titleComputers & Operations Researchen_US
dc.subjectExplainable artificial intelligenceen_US
dc.subjectFeature selectionen_US
dc.subjectStochastic approximationen_US
dc.subjectGradient descenten_US
dc.subjectBarzilai and Borwein methoden_US
dc.subjectGenetic algorithmen_US
dc.titleFeature selection using stochastic approximation with Barzilai and Borwein non-monotone gainsen_US
dc.typeArticleen_US

Files

Original bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
Feature_selection_using_stochastic_approximation_with_Barzilai_and_Borwein_non-monotone_gains__.pdf
Size:
880.64 KB
Format:
Adobe Portable Document Format
Description:
Download
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
1.69 KB
Format:
Item-specific license agreed upon to submission
Description:
Download

Collections

Department of Industrial Engineering