Assessment of Inter-examiner Agreement on Double Marking of Essay Papers

Abstract

Assessment method of medical students by conducting examination is to identify the quality and quantity of their academic performance. Essay paper is one of the most common assessment tools in the Faculty of Medicine and Health Sciences, Universiti Malaysia Sabah. Double marking is a means by which academic staff attempts to
produce fair results for the students. Eighty eight medical students sat for three sets of Essay Papers of Professional I examination in March 2012. The double marking on the essay papers was done by two lecturers of each clinical department concerned for each discipline. Inter-examiner agreement and its effect on the reliability of the final score for the students were calculated by using Kappa statistics and Intra-class Correlation
Coefficient (ICC). Reliability coefficient of the scores were also calculated for the different disciplines. In Part A essay paper, Cohan’s Kappa was 0.48 (p<0.001) and ICC was 0.943 (p<0.001) with Cronbach’s alpha = 0.95 for both markings. Pearson correlation was 0.91 (p<0.001). In Part B essay paper, Cohan’s Kappa was 0.28 (p<0.05) and ICC was 0.753 (p<0.001) with Cronbach’s alpha = 0.81 for both markings. Pearson correlation was 0.69 (p<0.001). In Part C essay paper, Cohan’s
Kappa was 0.02 (p>0.05) and ICC was 0.256 (p<0.001) with Cronbach’s alpha = 0.64 for both markings. Pearson correlation was 0.57 (p<0.001). Mean difference of double marking in Part A, Part B and Part C essay paper were −0.51 (SD= ±1.3), −1.11 (SD= ±1.4) and −5.25 (SD= ±2.6) respectively. There were more interrater reliability in Part A and Part B essay paper than Part C essay paper. Part A essay paper had more degree of consistency between raters than other papers as Pearson correlation was high. Mean difference of double marking in Part C essay paper was higher than other papers and degree of consistency between raters was also lower (Pearson correlation coefficient = 0.57). We conclude that there should be certain criteria which are designed carefully
and used with clear procedures which can reduce inconsistencies in assessment.