the Social Sciences 12.0 (SPSS, www.spss.com). Reliability of data sources as well as each
research question will be addressed here.
Interrater reliability. Reliability is defined as the degree to which multiple assessments
agree. When human beings are involved in making a judgment or a decision regarding a
measurement or outcome, some assessment of the reliability of the measurement needs to be
made. While the precise total scores given by the two graders on a given composition may differ,
good compositions should receive higher scores than average compositions and average
compositions higher than poor compositions. To test for this condition and determine interrater
reliability, the intraclass correlation coefficient or ICC (McGraw & Wong, 1996) was used.
The ICC is an ANOVA-type model in which the graders' composition scores are
responses. One source of variation is the student composition, which is assumed to be a random
sample from a larger pool of compositions. Another source of variation is the graders, who are
unique to this study and not from a random pool of graders. Thus, the two-way mixed effects
model was used. Moreover, since a similar (versus exact) pattern of scores is acceptable, it was
reasonable to check for consistency rather than absolute agreement. The competence and
reliability of the first rater were enhanced by his being a Spanish teacher with twenty years in the
profession, experienced in composition, and knowledgeable about national, university, and
departmental standards. He had not taught students in the test group. The second rater used in
this study was also a veteran Spanish teacher with twenty years in the profession at both the
secondary and university levels and experienced in teaching composition courses. She also had
not had any of the students in the test group. All compositions (N = 104) were used to determine
interrater reliability. The ICC for these two graders was .81.
Oxford, Raquel Malia Nitta. Effects of Technology-Enhanced Language Learning on Second Language Composition of University-Level Intermediate Spanish Students. Denton, Texas. UNT Digital Library. http://digital.library.unt.edu/ark:/67531/metadc4688/. Accessed May 5, 2015.