People display systematic affective reactions to specific properties of touched materials. For example, granular materials such as fine sand feel pleasant, while rough materials feel unpleasant. We wondered how far such relationships between sensory material properties and affective responses can be changed by learning. Manipulations in the present experiment aimed at unlearning the previously observed negative relationship between roughness and valence and the positive one between granularity and valence. In the learning phase, participants haptically explored materials that are either very rough or very fine-grained while they simultaneously watched positive or negative stimuli, respectively, from the International Affective Picture System (IAPS). A control group did not interact with granular or rough materials during the learning phase. In the experimental phase, participants rated a representative diverse set of 28 materials according to twelve affective adjectives. We found a significantly weaker relationship between granularity and valence in the experimental group compared to the control group, whereas roughness-valence correlations did not differ between groups. That is, the valence of granular materials was unlearned (i.e., to modify the existing valence of granular materials) but not that of rough materials. These points to differences in the strength of perceptuo-affective relations, which we discuss in terms of hard-wired versus learned connections.