We propose a method to recognize the scene of an image by finding the objects and the colors it contains. We approach this problem by creating a binary vector of detected objects and a histogram of the colors that the image contains. We then use these features to train a random forest classifier in order to determine the scene of each image. For class-based classifiers, our method gives comparable results with the state of art methods, such as Object Bank method, for the indoor scene dataset that we used. Additionally, while well-known methods are computationally expensive, our method has a low computational cost. © 2013 IEEE.