In this paper, a moving object tracking algorithm for infrared image sequences is presented. The tracking algorithm is based on the mean-shift tracking method which is based on comparing the histograms of moving objects in consecutive image frames. In video obtained after visible light, the color histogram of the object is used for tracking. In forward looking infrared image sequences, the histogram is constructed not only from the pixel values but also from a highpass filtered version of the original image. The reason behind the use of highpass filter outputs in histogram construction is to capture structural nature of the moving object. Simulation examples are presented.