In this paper, spectrum sensing in cognitive radio systems is studied for non-Gaussian channels in the presence of prior distribution uncertainty. In most practical cases, some amount of prior information about signals of primary users is available to secondary users but that information is never perfect. In order to design optimal spectrum sensing algorithms in such cases, we propose to employ the restricted Neyman-Pearson (NP) approach, which maximizes the average detection probability under constraints on the worst-case detection and false-alarm probabilities. We derive a restricted NP based spectrum sensing algorithm for additive Gaussian mixture noise channels, and compare its performance against the generalized likelihood ratio test (GLRT) and the energy detector. Simulation results show that the proposed spectrum sensing algorithm provides improvements over the other approaches in terms of minimum (worst-case) and/or average detection probabilities. © 2013 IEEE.