Friction on few-layer graphene is known to exhibit unique layer dependence where friction measured via atomic force microscopy (AFM) on the nanometer scale is generally observed to decrease with increasing number of layers. However, this trend is not always observed for AFM probe tips with different sizes and for graphene on different substrates. Within this context, the precise role played by the interface, in particular, the size of the contact and substrate roughness, in the layer dependence of friction on graphene is not yet completely understood. Here, we probe the origins of the roughness dependence of layer-dependent friction on graphene by a combination of AFM measurements and molecular dynamics (MD) simulations. In the experiments, friction is observed to monotonically decrease with increasing number of graphene layers for tips with various apex radii, while the roughness of the sample surface is observed to decrease. In the simulations, two opposite layer-dependence trends for friction are observed on few-layer graphene on substrates with different roughness values. The underlying mechanisms are investigated using atomistic details obtained from the simulations, where the different friction trends are found to originate from an interplay between surface roughness, the trajectory of the tip, and the number of atoms in contact. Finally, the effect of topographical correlation length on the layer dependence of friction on graphene is discussed.