Analytic Rubric Format: How Category Position Affects Raters' Mental Rubric

ICC Auditorium & ICC Room 115

Presented at: ECOLT 2017

Analytic rubrics present detailed criteria descriptors listed by category and score band (Barkaoui, 2011; Weigle, 2002). Because of the potentially long, detailed category-by-category layout, analytic rubrics could trigger ordering effects, in which raters pay less attention to category descriptors (or entire categories) as they read from left to right or top to bottom.  The large number of descriptors included in analytic rubrics may cause a heavy cognitive load, which can become unmanageable for raters (Hamp-Lyons & Henning, 1991).  If raters do not have the attention or capacity to equally attend to all rubric descriptors, then information-ordering effects could be to blame and may cause an unequal representation of categories in raters’ mental rubric. 

In this study, I investigate whether analytic rubric format (i.e. category order) affects raters’ mental representation of the rubric categories. In a within-subjects counter-balanced design, 31 novice raters were randomly assigned to two groups and were trained on two rubrics in two rounds.  The analytic rubrics were identical in content, but varied in the order the categories appeared. In Round 1, raters trained on one rubric and rated 20 essays. Five weeks later, in Round 2, raters trained on the alternate rubric and re-rated the 20 essays.  Throughout the study, I measured raters’ recall of category descriptors and raters’ estimation of descriptor importance.  I performed repeated-measure ANOVAs to uncover differences in rater’s category recall and category importance between the two rubrics.

Results show that category position affected 1) the raters’ beliefs about what criteria are the most and least important when scoring an essay, and 2) how many descriptors raters were able to recall from a category.  Raters favored the category that appeared first on the rubric. Based on these findings, I discuss rater training, rubric design, and test-construct considerations for rubric designers and test developers.