Using DIF Analysis to Identify Linguistic Features that Yield Differential Performance Among Language Groups

-

Please consult the event program for date, time and location.

Presented at: AAAL 2015

The construction of a large-scale assessment entails many important considerations, including the assurance that all items are unbiased and do not favor one group over another. This is traditionally evaluated using differential item functioning (DIF) analysis, which compares groups to determine whether people with the same underlying ability have a different probability of giving a certain response “because of some characteristic of the test item and/or testing situation”(Zumbo, 2007, p.229). For English Language Proficiency (ELP) examinations, test takers are conventionally evaluated for DIF using ethnicity (i.e., comparing Hispanics and non-Hispanics) as a proxy for home language because states employ a variety of methods of coding home language and home language information is often not reported by caregivers. This paper investigates this practice by assessing whether DIF analysis of ethnicity and home language offer similar results across test items and identifying contexts where this substitution might fall short.

To evaluate these questions, this paper examines test items from the ACCESS for ELLs® ELP assessment (Center for Applied Linguistics, 2012), Items are analyzed using Coh-Metrix (McNamara, Louwerse, Cai, & Graesser, 2005) to quantify such linguistic features as lexical complexity, syntactic complexity, cohesion, and coherence, which might affect the responses of different language groups. The linguistic features are normalized against the length of each item, and items are bundled together based on their linguistic characteristics to determine whether particular linguistic features influence student performances across different home language groups. Based on the close linguistic relationship that English shares with Germanic and Romance languages (Finkenstaedt & Wolff, 1973; Williams, 1986), speakers of languages from these families are compared with students whose first languages are more distant from English. Results are of particular importance for developers of high-stakes tests, it is imperative that such assessments allow students of all backgrounds to demonstrate their language proficiency.