This paper proposes a video-based fire detection system which uses color, spatial and temporal information. The system divides the video into spatio-temporal blocks and uses covariance-based features extracted from these blocks to detect fire. Feature vectors take advantage of both the spatial and the temporal characteristics of flame-colored regions. The extracted features are trained and tested using a support vector machine (SVM) classifier. The system does not use a background subtraction method to segment moving regions and can be used, to some extent, with non-stationary cameras. The computationally efficient method can process 320×240 video frames at around 20 frames per second in an ordinary PC with a dual core 2.2 GHz processor. In addition, it is shown to outperform a previous method in terms of detection performance.