An assessment of the content validity of the midterm achievement tests administered at Anadolu University Foreign Languages Department

Abstract

The purpose of this study was to analyze the content validity o f the first semester midterm tests in the Foreign Languages Department (FLD) at Anadolu University. Content validation involves the systematic analysis of test content to determine whether it covers a representative sample of what has been taught in a specific course. This analysis can be done through using multiple sources. Teachers’ opinions about the validity of the tests can be elicited, the syllabus content can be compared to the test content and the course objectives can be compared to the test content. Use of these multiple sources ensures the soundness of the analysis and strengthens any arguments to be made about validity. To investigate the content validity of the midterm tests in the FLD, teachers’ perceptions o f the tests’ representation of the classroom material content were elicited by using uestionnaires. All of the course materials at the intermediate level and the two midterm tests from each course were analysed for their content and task frequencies and the correlations of these were computed using Pearson Product Moment Correlation. To learn the objectives of the courses, the coordinators responsible for each course were interviewed. The data from each of these three sources were then compared in order to assess the content validity o f the tests. The results were conflicting. The instructors of the listening, core course and grammar courses thought their representation of the course content on the test was moderate to high on the whole. Though they were able to justify their exclusion of some items on the tests, which indicated that they actually had some objectives, these were not clearly stated in a written form, a very important point for establishing test validity. The material-test frequency results for these courses indicated that; except for the first grammar midterm test and the strategy areas of the second listening midterm test, the material and tests had low correlations, especially in terms of exercise types. The analysis o f the course objectives also showed that they were not specific enough and their overall agreement with the tests’ content was low. If the relevant literature is taken into account, the conflict among the results can be said to have resulted from teachers’ not having clear objectives and test specifications in their test design. Finally, the following suggestions were made to improve the testing procedures in the FLD at Anadolu University: First, course objectives should be clarified and refined. Second, explicit test specifications should be prepared. For further improvement of the validity of the tests, establishing a testing office, which would focus only on testing issues, can also be considered by the FLD administration. Implications for further study were discussed.