The Influence of Changes in Assessment Design on the Psychometric Quality of Scores

In this article, we address the problem of improving the measurement quality of a complex performance assessment through principled assessment design. We describe the characteristics and measurement impact of steps taken to improve assessment exercise design along with modifications in assessor training materials and procedures between the 1995-1996 and the 1996-1997 administrations of the National Board for Professional Teaching Standards Early Childhood/Generalist examination. Specifically, we describe how the revision of this assessment contributed to increases in the interassessor agreement, internal consistency, and generalizability of scores. All indexes we examined improved after the revisions. The results suggest that previously observed limits on the measurement quality of performance assessments due to the relatively small number of items that contribute to an assessment score may be altered significantly through attention to assessment design and related scoring processes.