This article reports on a study of the validity and reliability of tests administered in an EFL university setting. The study addresses the question of how well face validity reflects more objective measures of the quality of a test, such as predictive validity and reliability. According to some researchers, face validity, defined as the surface credibility or public acceptability of a test, has no theoretical basis since it is based on the subjective perceptions of stakeholders such as teachers and students. However, due to lack of time or resources, or due to a perceived lack of competence, practitioners tend to rely on the ‘appeal’ of language tests, rather than seek empirical evidence. This article describes several ways of evaluating achievement tests, comparing their results in order to shed light on what measures can and should be taken to ensure that achievement tests accomplish their purposes.