Fractional fourier transform meets transformer encoder

Date

2022-10-28

Editor(s)

Advisor

Supervisor

Co-Advisor

Co-Supervisor

Instructor

Source Title

IEEE Signal Processing Letters

Print ISSN

1070-9908

Electronic ISSN

1558-2361

Publisher

Institute of Electrical and Electronics Engineers

Volume

29

Issue

Pages

2258 - 2262

Language

English

Journal Title

Journal ISSN

Volume Title

Series

Abstract

Utilizing signal processing tools in deep learning models has been drawing increasing attention. Fourier transform (FT), one of the most popular signal processing tools, is employed in many deep learning models. Transformer-based sequential input processing models have also started to make use of FT. In the existing FNet model, it is shown that replacing the attention layer, which is computationally expensive, with FT accelerates model training without sacrificing task performances significantly. We further improve this idea by introducing the fractional Fourier transform (FrFT) into the transformer architecture. As a parameterized transform with a fraction order, FrFT provides an opportunity to access any intermediate domain between time and frequency and find better-performing transformation domains. According to the needs of downstream tasks, a suitable fractional order can be used in our proposed model FrFNet. Our experiments on downstream tasks show that FrFNet leads to performance improvements over the ordinary FNet.

Course

Other identifiers

Book Title

Citation