EFL composition assessment: the relationships between teacher-rater background characteristics and reader reliability

Abstract

It is common knowledge among composition researchers that different composition readers judge the same compositions differently due to certain reader variables. This controversial issue leads to a further problem, which is best termed as the reader reliability and unreliability dichotomy. While it is assumed that some ccatposition rater background characteristics influence reader reliability, few studies have investigated the relationships between rater background characteristics and reader reliability. This research study explored the relationships between composition teacher-rater background characteristics and reader reliability at the Eastern Mediterranean University Preparatory School in the Turkish Republic of Northern Cyprus. A background information questionnaire consisting of fourteen items was first developed, and then administered to 47 teacher-raters who had scored an EFL composition achievement test consisting of one prompt. The research participants scored a total number of 1191 student compositions. The arrangement of the scoring process was such that two teachers independently scored a subset of about 30 compositions using a twenty-point analytic scale. The researcher was also provided with the composition assessment record prior to the administration of the questionnaire. The two sets of scores cissigned to the compositions in each subset were correlated in order to identify the reliable and the xmreliable readers within pairs. At this point, the dependent reliability variable was created so that it could be regressed with the independent background characteristics variables. One of the 14 independent variables — participation in reader group-training sessions before scoring English composition examination papers — was excluded from the stxody, and the analysis was done with 13 independent variables. This variable was excluded because all of the raters' responses were "yes" to this background characteristic. The analysis did not yield any statistically significant relationships between the depaident variable and the independent variables group. Therefore, the alternative hypothesis stating that there are EFL canrposition teacher-rater background characteristics related to reader reliability was rejected, and the null hypothesis stating that there are no EFL corrposition teacher-rater background characteristics related to reader reliability had to be accepted. Only one rater out of the 47 raters was identified ëis unreliable. However, it is believed that factors other than background characteristics, such as fatigue, physical distractions, or subconscious rebellious behavior against the testing administrators affected his/her reading performance. This conclusion was supported by some collected data and insitu observations.