I suggest that the brain’s low-level sensory-motor systems are organized on the basis of prediction. This suggestion differs radically from existing theories of sensory-motor systems, and can be summarized as follows. Certain simple mechanisms in the brain predict the current or future states of other brain mechanisms. These mechanisms can be established and disposed dynamically. Successful prediction acts as a kind of selection criteria .and new structures are formed and others are disposed according to their predictive powers. Simple mechanisms become connected to each other on the basis of their predictive power, possibly establishing hierarchical structures, and forming large complexes. The complexes so formed, can implement a number of functionalities including detecting interesting events, creating high-level representations, and helping with goal-directed activity. Faculties such as attention and memory contribute to such processes of internal predictions and they can be studied and understood within this setting. All of this does not rule out the existence of other mechanisms, but an organization driven by prediction serves as the backbone of low-level cognitive activity. I develop a computational model of a sensory-motor system that works on this basis. I also show how this model explains certain interesting aspects of human perception and how it can be related to general cognitive capabilities.