Energy-Efficient LSTM networks for online learning

Simple item page
buir.contributor.authorMirza, Ali H.
buir.contributor.authorKozat, Süleyman Serdar
dc.citation.epage3126en_US
dc.citation.issueNumber8en_US
dc.citation.spage3114en_US
dc.citation.volumeNumber31en_US
dc.contributor.authorErgen, T.
dc.contributor.authorMirza, Ali H.
dc.contributor.authorKozat, Süleyman Serdar
dc.date.accessioned2021-02-18T11:12:44Z
dc.date.available2021-02-18T11:12:44Z
dc.date.issued2020
dc.departmentDepartment of Electrical and Electronics Engineeringen_US
dc.description.abstractWe investigate variable-length data regression in an online setting and introduce an energy-efficient regression structure build on long short-term memory (LSTM) networks. For this structure, we also introduce highly effective online training algorithms. We first provide a generic LSTM-based regression structure for variable-length input sequences. To reduce the complexity of this structure, we then replace the regular multiplication operations with an energy-efficient operator, i.e., the ef-operator. To further reduce the complexity, we apply factorizations to the weight matrices in the LSTM network so that the total number of parameters to be trained is significantly reduced. We then introduce online training algorithms based on the stochastic gradient descent (SGD) and exponentiated gradient (EG) algorithms to learn the parameters of the introduced network. Thus, we obtain highly efficient and effective online learning algorithms based on the LSTM network. Thanks to our generic approach, we also provide and simulate an energy-efficient gated recurrent unit (GRU) network in our experiments. Through an extensive set of experiments, we illustrate significant performance gains and complexity reductions achieved by the introduced algorithms with respect to the conventional methods.en_US
dc.description.sponsorshipThis work was supported in part by the Tubitak Project under Grant 117E153.en_US
dc.identifier.doi10.1109/TNNLS.2019.2935796en_US
dc.identifier.issn2162-237X
dc.identifier.urihttp://hdl.handle.net/11693/75451
dc.language.isoEnglishen_US
dc.publisherIEEEen_US
dc.relation.isversionofhttps://dx.doi.org/10.1109/TNNLS.2019.2935796en_US
dc.source.titleIEEE Transactions on Neural Networks and Learning Systemsen_US
dc.subjectEf-operatoren_US
dc.subjectExponentiated gradient (EG)en_US
dc.subjectGradient descenten_US
dc.subjectLong short-term memory (LSTM)en_US
dc.subjectMatrix factorizationen_US
dc.titleEnergy-Efficient LSTM networks for online learningen_US
dc.typeArticleen_US
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
Energy-Efficient_LSTM_Networks_for_Online_Learning.pdf
Size:
1.89 MB
Format:
Adobe Portable Document Format
Description:
Download
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
1.71 KB
Format:
Item-specific license agreed upon to submission
Description:
Download
Collections
Department of Electrical and Electronics Engineering