Fractional fourier transform meets transformer encoder

Simple item page
buir.contributor.authorŞahinuç, Furkan
buir.contributor.authorKoç, Aykut
buir.contributor.orcidŞahinuç, Furkan|0000-0001-9104-2860
buir.contributor.orcidKoç, Aykut|0000-0002-6348-2663
dc.citation.epage2262en_US
dc.citation.spage2258en_US
dc.citation.volumeNumber29en_US
dc.contributor.authorŞahinuç, Furkan
dc.contributor.authorKoç, Aykut
dc.date.accessioned2023-02-28T13:30:10Z
dc.date.available2023-02-28T13:30:10Z
dc.date.issued2022-10-28
dc.departmentDepartment of Electrical and Electronics Engineeringen_US
dc.departmentNanotechnology Research Center (NANOTAM)en_US
dc.description.abstractUtilizing signal processing tools in deep learning models has been drawing increasing attention. Fourier transform (FT), one of the most popular signal processing tools, is employed in many deep learning models. Transformer-based sequential input processing models have also started to make use of FT. In the existing FNet model, it is shown that replacing the attention layer, which is computationally expensive, with FT accelerates model training without sacrificing task performances significantly. We further improve this idea by introducing the fractional Fourier transform (FrFT) into the transformer architecture. As a parameterized transform with a fraction order, FrFT provides an opportunity to access any intermediate domain between time and frequency and find better-performing transformation domains. According to the needs of downstream tasks, a suitable fractional order can be used in our proposed model FrFNet. Our experiments on downstream tasks show that FrFNet leads to performance improvements over the ordinary FNet.en_US
dc.description.provenanceSubmitted by Ayça Nur Sezen (ayca.sezen@bilkent.edu.tr) on 2023-02-28T13:30:10Z No. of bitstreams: 1 Fractional_fourier_transform_meets_transformer_encoder.pdf: 452791 bytes, checksum: 6d69ae0dfcd227c35dbf874c229f06af (MD5)en
dc.description.provenanceMade available in DSpace on 2023-02-28T13:30:10Z (GMT). No. of bitstreams: 1 Fractional_fourier_transform_meets_transformer_encoder.pdf: 452791 bytes, checksum: 6d69ae0dfcd227c35dbf874c229f06af (MD5) Previous issue date: 2022-10-28en
dc.identifier.doi10.1109/LSP.2022.3217975en_US
dc.identifier.eissn1558-2361
dc.identifier.issn1070-9908
dc.identifier.urihttp://hdl.handle.net/11693/111947
dc.language.isoEnglishen_US
dc.publisherInstitute of Electrical and Electronics Engineersen_US
dc.relation.isversionofhttps://doi.org/10.1109/LSP.2022.3217975en_US
dc.source.titleIEEE Signal Processing Lettersen_US
dc.subjectEncoderen_US
dc.subjectFNeten_US
dc.subjectFourier transformen_US
dc.subjectFractional fourier transformen_US
dc.subjectTransformeren_US
dc.titleFractional fourier transform meets transformer encoderen_US
dc.typeArticleen_US

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
Fractional_fourier_transform_meets_transformer_encoder.pdf
Size:
442.18 KB
Format:
Adobe Portable Document Format
Description:
Download
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
1.69 KB
Format:
Item-specific license agreed upon to submission
Description:
Download

Collections

Department of Electrical and Electronics Engineering
Nanotechnology Research Center (NANOTAM)