Peeling Back the Onion: Developing a Validity Argument for an Extant Assessment

Please consult the event program for time and location.

Presented at: ECOLT 2014

Given the increasing role of student assessment in the United States education system, there has been a growing awareness within the language testing community that assessments must be designed with their interpretation and consequences in mind. A number of argument-based approaches, like Interpretive Arguments (e.g., Chapelle, Enright, & Jamieson, 2008; Kane, 2002, 2013) and Assessment Use Arguments (Bachman & Palmer, 2010), present frameworks that are used for evaluating the validity of the use of an assessment for its intended purposes. A significant shift from the traditional approach to validity, which presented different types of validity evidence without an overarching theme or narrative, argument-based methods contextualize validity data to increase the interpretability and accessibility of technical information so that it is available to a broader audience. Through this sort of lens, both test developers and stakeholders are able to identify whether there is evidence that supports the appropriateness and adequacy of the interpretations and decisions made regarding examinees on the basis of their test performance.

While argument-based methods have been constructed to evaluate the validity of new assessments, like the revised Test of English as a Foreign Language TM Internet-based Test in 2005 (Chapelle, Enright, & Jamieson, 2008), this paper describes the process of developing this type of framework for an existing large-scale English language assessment. In particular, it will highlight the challenges of beginning to create a multi-layered argument-based structure and adapting it to a language assessment that is currently in use. In addition, it will discuss many of the lessons that were learned during the process of building the validity framework, including determining a starting point for constructing an argument, highlight the benefits of increasing accessibility and interpretability for stakeholders, and suggest future plans for development and expansion.