We investigate the computation of a sparse solution to an underdetermined system of linear equations using the Huber loss function as a proxy for the 1-norm and a quadratic error term à la Lasso. The approach is termed “penalized Huber loss”. The results of the paper allow to calculate a sparse solution using a simple extrapolation formula under a sign constancy condition that can be removed if one works with extreme points. Conditions leading to sign constancy, as well as necessary and sufficient conditions for computation of a sparse solution by penalized Huber loss, and ties among different solutions are presented.