Deep learning based channel equalization for MIMO ISI channels

Future wireless communications is expected to bring significant changes along with a number of emerging technologies such as 5G, virtual reality, edge computing, and IoT. These developments pose unprecedented demands in terms of capacity, coverage, latency, efficiency, flexibility, compatibility, and quality of experience on wireless communication systems. Machine Learning (ML) techniques are considered as a promising tool to tackle this challenge due to their ability to manage big data, powerful nonlinear mapping, and distributed computing capabilities. There have been many research results addressing different aspects of ML algorithms and their connections to wireless communications; however, there are still various challenges that need to be addressed. In particular, their use for communication systems with memory, is not fully investigated. With this motivation, this thesis considers an application of ML, in particular, deep learning (DL), techniques for communications over intersymbol interference (ISI) channels. In this thesis, we propose DL-based channel equalization algorithms for channels with ISI. We introduce three different DL-based ISI detectors, namely sliding bidirectional long short term memory (Sli-BiLSTM), sliding multi layer perceptron (Sli-MLP), and sliding iterative (Sli-Iterative), and demonstrate that they are computationally efficient and capable of performing equalization under a variety of channel conditions with the knowledge of the channel state information. We also employ sliding bidirectional gated recurrent unit (Sli-BiGRU) and Sli- MLP, which are more suitable for use with fixed ISI channels. As an extension, we also examine DL-based equalization techniques for multiple-input multipleoutput (MIMO) ISI channels. Numerical results show that proposed models are well suited for equalization of ISI channels with perfect as well as noisy CSI for a broad range of signal-to-noise ratio (SNR) levels as long as the ISI length is very close to the optimal solution, namely, the maximum likelihood sequence estimation, implemented through the Viterbi Algorithm while having considerably less complexity, and they have superior performance compared to MMSE-based channel equalization.