Fractional fourier transform meets transformer encoder

Date
2022-10-28
Editor(s)
Advisor
Supervisor
Co-Advisor
Co-Supervisor
Instructor
Source Title
IEEE Signal Processing Letters
Print ISSN
1070-9908
Electronic ISSN
1558-2361
Publisher
Institute of Electrical and Electronics Engineers
Volume
29
Issue
Pages
2258 - 2262
Language
English
Journal Title
Journal ISSN
Volume Title
Series
Abstract

Utilizing signal processing tools in deep learning models has been drawing increasing attention. Fourier transform (FT), one of the most popular signal processing tools, is employed in many deep learning models. Transformer-based sequential input processing models have also started to make use of FT. In the existing FNet model, it is shown that replacing the attention layer, which is computationally expensive, with FT accelerates model training without sacrificing task performances significantly. We further improve this idea by introducing the fractional Fourier transform (FrFT) into the transformer architecture. As a parameterized transform with a fraction order, FrFT provides an opportunity to access any intermediate domain between time and frequency and find better-performing transformation domains. According to the needs of downstream tasks, a suitable fractional order can be used in our proposed model FrFNet. Our experiments on downstream tasks show that FrFNet leads to performance improvements over the ordinary FNet.

Course
Other identifiers
Book Title
Citation
Published Version (Please cite this version)