This paper presents a novel infrared (IR) object tracking algorithm based on the co-difference matrix. Extraction of co-difference features is similar to the well known covariance method except that the vector product operator is redefined in a multiplication-free manner. The new operator yields a computationally efficient implementation for real time object tracking applications. Experiments on an extensive set of IR image sequences indicate that the new method performs better than covariance tracking and other tracking algorithms without requiring any multiplication operations.