The restricted Neyman–Pearson (NP) approach is studied for composite hypothesis-testing problems in the presence of uncertainty in the prior probability distribution under the alternative hypothesis. A restricted NP decision rule aims to maximize the average detection probability under the constraints on the worst-case detection and false-alarm probabilities, and adjusts the constraint on the worst-case detection probability according to the amount of uncertainty in the prior probability distribution. In this study, optimal decision rules according to the restricted NP criterion are investigated. Also, an algorithm is provided to calculate the optimal restricted NP decision rule. In addition, it is shown that the average detection probability is a strictly decreasing and concave function of the constraint on the minimum detection probability. Finally, a detection example is presented to investigate the theoretical results, and extensions to more generic scenarios are provided.