Optimal stochastic signaling under average power and bit rate constraints

Abstract

The optimal stochastic signaling based on the joint design of prior distribution and signal constellation is investigated under an average bit rate and power constraints. First, an optimization problem is formulated to maximize the average probability of correct decision over the set of joint distribution functions for prior probabilities and the corresponding constellation symbols. Next, an alternative problem formulation, for which the optimal joint distribution is characterized by a randomization among at most three mass points, is provided, and it is shown that both formulations share the same solution. Three special cases of the problem are investigated in detail. First, in the absence of randomization, the optimal prior probability distribution is analyzed for a given signal constellation and a closed-form solution is provided. Second, the optimal deterministic pair of prior probabilities and the corresponding signal levels are considered. Third, a binary communication system with scalar observations is investigated in the presence of a zero-mean additive white Gaussian noise, and the optimal solution is obtained under practical assumptions. Finally, numerical examples are presented to illustrate the theoretical results. It is observed that the proposed approach can provide improvements in terms of average symbol error rate over the classical scheme for certain scenarios.