Deep-learning for communication systems: new channel estimation, equalization, and secure transmission solutions

Abstract

Traditional communication system design takes a model-based approach that aims to optimize relevant performance metrics using somewhat simple and tractable channel and signal models. For instance, channel codes are designed for simple additive white Gaussian or fading channel models, channel equalization algorithms are based on mathematical models for inter-symbol interference (ISI), and channel estimation techniques are developed with the underlying channel statistics and characterizations in mind. Through utilizing superior mathematical models and expert knowledge in signal processing and information theory, the model-based approach has been highly successful and has enabled development of many communication systems until now. On the other hand, beyond 5G wireless communication systems will further exploit the massive number of antennas, higher bandwidths, and more advanced multiple access technologies. As communication systems become more and more complicated, it is becoming increasingly important to go beyond the limits of the model-based approach. Noting that there have been tremendous advancements in learning from data over the past decades, a major research question is whether machine learning based approaches can be used to develop new communication technologies. With the above motivation, this thesis deals with the development of deep neural network (DNN) solutions to address various challenges in wireless communications. We first consider orthogonal frequency division multiplexing (OFDM) over rapidly time-varying multipath channels, for which the performance of standard channel estimation and equalization techniques degrades dramatically due to inter-carrier in-terference (ICI). We focus on improving the overall system performance by designing DNN architectures for both channel estimation and data demodulation. In addition, we study OFDM over frequency-selective channels without cyclic prefix insertion in an effort to improve the overall throughputs. Specifically, we design a recurrent neu-ral network to mitigate the effects of ISI and ICI for improved symbol detection. Furthermore, we explore secure transmission over multi-input multi-output multi-antenna eavesdropper wiretap channels with finite alphabet inputs. We use a linear precoder to maximize the secrecy rate, which benefits from the generalized singular value decomposition to obtain independent streams and exploits function approximation abilities of DNNs for solving the required power allocation problem. We also propose a DNN technique to jointly optimize the data precoder and the power allocation for artificial noise. We use extensive numerical examples and computational complexity analyses to demonstrate the effectiveness of the proposed solutions.