Developing Innovative Item Types for a Large-Scale Assessment of English Language Proficiency

-
Presented at: Conference on Language, Learning and Culture: Next Generation Assessment

The development and expansion of technology in recent decades has affected education in myriad ways, including how students’ language proficiency can be measured. As English language proficiency (ELP) educational assessments for grades Kindergarten (K)–12 English language learner (ELL) students move from paper to computer-based administration methods, it is imperative to: (a) create the most effective testing environment for students, particularly the youngest learners; (b) increase engagement and user focus; and (c) effectively measure the intended construct.

One way to achieve these goals is by integrating enhanced multi-modal input and new item types, such as drag and drop and hotspot, into assessments that have traditionally used multiple choice items. Drag and drop and hotspot item types allow for text and graphic stimuli to be contextualized in more authentic presentation formats which are more engaging to students. In addition, the ability to provide multi-modal input for student directions and task support helps ensure that ELL students are best able to demonstrate their language proficiency.

This poster will discuss key questions and findings from new item development and cognitive interview studies on these new item types to share insights into how these innovative item types are being designed and refined to enhance K–12 ELP assessments. Capitalizing on the opportunities offered by technology to improve ELP assessments requires purposeful planning and study to provide evidence that new item types assess the given construct effectively without introducing construct-irrelevant interference. As previous studies have shown, innovative item types, when implemented well, have the potential to gather more information than multiple choice items about students at all ability levels (Jodoin, 2003). In addition, innovative item types “can greatly enhance the validity of interpretations made on the basis of test scores” (Sireci & Zenisky, 2006, p.329).

As part of the research and development phase of test development for ACCESS for ELLs 2.0, a computer-delivered large scale K–12 ELP test, cognitive labs are carried out with the target population to inform the prototyping process and the development of specifications. These cognitive labs include observation and interview data which inform continued improvement of test items and ensure that test users are interacting with test items as expected. Both the process of developing new item types for an extant ELP assessment and preliminary findings from the cognitive labs will be presented, highlighting the challenges and possibilities that accompany the use of computer-based technologies in large-scale assessments for ELLs.