English Education Professor Young-Ju Lee

논문 초록 (Abstract)

Myford & Wolfe’s (2003, 2004) papers suggested how multi-faceted Rasch analysis (using Facets) might be used to illustrate the effectiveness and validity of various statistical indicators in detecting and measuring a number of different rater effects using data from judgement-based contexts. While the arguments presented by Myford and Wolfe were clearly of significance for measurement specialists, this study sets out to explore how practically relevant they might be for classroom teachers as well as test developers. This study explores a dataset containing the ratings by a group of 20 raters of 81 writing scripts from a major international English language examination. The purpose of the study was to see to what extent the statistical indicators that Myford & Wolfe proposed were useful for identifying problematic raters in the real-world data set. To do so, this study compared group- and individual-level indicators of three rater effects (severity/ leniency, randomness, and halo effect) between Myford and Wolfe’s simulated data sets and the real-world data set. Rater behavior was modeled using the Facets program, Version 3.64 (Linacre, 2008). A comparison of results showed that the indicators suggested by Myford & Wolfe proved as effective with real data as they had done with simulated data. The implication of this finding is that the use of MFRM using Facets is likely to offer a valuable statistical tool for the exploration of rater effects not only in large-scale test systems but also in classroom assessments. Findings of this study can provide practical implications for English teachers in middle and high schools who will be conducting a small-scale study of rater effects in performance assessments.