Download a PDF of this digest.

Abstract This digest reports on a descriptive study examining two questions related to adult English language learners’ educational level gains in the National Reporting System for Adult Education (NRS), as measured by the standardized oral proficiency assessment BEST Plus: (1) What is the relationship between instructional hours and educational level gain on BEST Plus? and (2) What is the relationship between intensity of instruction and educational level gain on BEST Plus? To provide guidance to users on the number of instructional hours needed for students to show a level gain on BEST Plus, the Center for Applied Linguistics collected pretest and posttest data on more than 6,500 students from two states. The largest number of students pretested with BEST Plus tested into the NRS Beginning ESL Literacy level (49%). The fewest were placed in the NRS Advanced ESL level (7%). Statistical analyses were conducted to see how examinee performance at each level varied according to number of instructional hours and intensity of instruction. Instructional hours were defined as the number of hours that students actually attended class, ranging from fewer than 60 hours to more than 140 hours. Intensity of instruction was defined as how often students attended class over a given period of time, ranging from low intensity (e.g., 100 hours of instruction over 250 days) to high intensity (e.g., 100 hours of instruction over 75 days). Results showed that across NRS educational functioning levels, the greater the number of instructional hours, the higher the percentage of students who made level gain. There was also a general trend toward greater NRS level gain for students with high levels of instructional intensity than for those with low intensity. Intensity of instruction had the greatest impact on students at the Beginning ESL Literacy, Low Intermediate, and Advanced ESL levels. Additional research is needed to better understand differences in level gain and the learner and programmatic factors, including number of instructional hours and intensity of instruction, that may influence oral language proficiency as measured by NRS level gain.

 

Defining Adult English Language Learners’ Educational Level Gain

The National Reporting System for Adult Education (NRS) is the accountability system for federally funded adult basic education, adult secondary education, and English as a second language (ESL) education in the United States. (See www.nrsweb.org for more information.) Upon enrollment, students in these programs place into one of six educational functioning levels based on their pretest scores. Their progress through these levels is reported each year by state departments of education to the U.S. Department of Education, Office of Vocational and Adult Education. Each state negotiates a target percentage of students at each educational functioning level that will advance at least one level (educational level gain) each year. Educational level gain in language and literacy is measured by pretesting students with an approved standardized assessment, then posttesting them with an equivalent form of the same assessment after a predetermined number of instructional hours or at the end of an instructional cycle. The minimum number of instructional hours recommended between pretesting and posttesting for NRS-approved assessments ranges from 40 to 120 hours (National Reporting System, 2006).

Adults learning English as a second language are placed in NRS ESL levels that range from Beginning ESL Literacy to Advanced ESL. The primary objective of many of these learners is to develop the English language and literacy skills they need to meet their personal, community, academic, and employment goals (National Center for ESL Literacy Education, 2003). Adult ESL programs focus on helping students develop the oral communication and literacy skills they need to attain their goals, which may include helping their children in school, obtaining a high school diploma, entering a postsecondary education program, getting a better job, and earning U.S. citizenship.

Learner and Program Factors Related to the Rate of English Acquisition by Adult English Language Learners

In 2003-2004 (the year for which the most recent data are available on students in federally funded programs), 1,172,579 students were enrolled in federally funded adult ESL classes; 36% of these students attained an educational level gain after a course of instruction (U.S. Department of Education, Office of Vocational and Adult Education, 2006). Given the variety of instructional, environmental, and personal factors that affect second language acquisition, it is difficult to say what factors might have the greatest impact on how quickly adult ESL students make gains in learning English. Adult learners come to ESL programs with a variety of prior educational and life experiences, English language proficiency levels, and educational goals. Learners also differ in their opportunities for language acquisition outside the classroom and in the circumstances that affect their participation in class. Some may work in jobs that require them to use English, whereas others may not. Some learners are able to attend ESL classes several times a week and have opportunities to study English outside the program, while others do not. Other factors such as language aptitude, age, and motivation may also play a part in rates of English language acquisition.

Specific program factors may also affect learners’ educational gains. Adult education programs are often tailored to take advantage of the few hours (typically 4 to 8 hours per week) that adult learners are available to study. Instruction may focus on a limited number of learner goals (e.g., finding a better job or helping children with their homework) or may span a wide variety of content topics and language skills. It is important to keep all of these factors in mind when considering expectations for adult ESL educational gain and attainment of goals. See Table 1 for an overview of these factors. Moss and Ross-Feldman (2003) offer information about how to approach English language acquisition in adult students through an understanding of what the research says about learner motivation (e.g., Dörnyei, 2002), interaction (e.g., Long, 1996), task-based learning (e.g., Ellis, 2000), and vocabulary (e.g., Coady & Huckin, 1997).

Table 1: Learner and Program Factors Affecting Adult English Language Acquisition and Instruction
Learner Factors	Program Factors
Time available for instruction Opportunities to use English outside the classroom Age Language and cultural background Level of prior education Degree of first language literacy Personal conditions (e.g., visual or auditory impairment) Personal motivation Other demands of life (e.g., work, family)	Instructional setting (e.g., school, work site) Type of instruction (e.g., classroom, tutoring) Type of entry (open vs. managed enrollment) Time of instruction (day, evening, weekend) Length of instructional program Level of resources
From Teachers of English to Speakers of Other Languages (2003)

 

BEST Plus Educational Level Gain Study

BEST Plus is one of the standardized assessments approved for accountability reporting for the NRS. BEST Plus assesses the oral proficiency of adult English language learners across three components:

• Listening comprehension measures how well students understand spoken English.
  
  
• Language complexity measures the depth and breadth of students’ use of conversational English.
  
  
• Communication measures how well students can be understood when speaking English.

Following a complete test administration, a score report is generated that expresses the final test results in terms of a BEST Plus scale score and a Student Performance Level (SPL). (See www.cal.org/caela/esl_resources/slspls.html for SPL descriptors.) BEST Plus scale scores are also aligned with the NRS ESL educational functioning levels (see Table 2).

Table 2. Alignment of NRS Levels, Student Performance Levels, and BEST Plus Scale Scores
NRS Level	SPL	BEST Plus Score Range
Beginning ESL Literacy	0-1	400 and below
Low Beginning ESL	2	401 to 417
High Beginning ESL	3	418 to 438
Low Intermediate ESL	4	439 to 472
High Intermediate ESL	5	473 to 506
Advanced ESL	6	507 to 540
Exit from NRS ESL levels	7	541 and higher

Since the launch of BEST Plus in 2003, many users have expressed interest in knowing the number of instructional hours needed for learners to show a level gain on this assessment. To provide some guidance to users on this issue, the Center for Applied Linguistics collected pretest and posttest data from two states and conducted statistical analyses to see how examinee performance at each of the NRS educational functioning levels varied according to number of instructional hours and intensity of instruction. Two research questions were addressed:

Question 1: What is the relationship between instructional hours and educational level gain on BEST Plus?

Question 2: What is the relationship between intensity of instruction and educational level gain on BEST Plus?

Background Information

BEST Plus pretest and posttest scores (for tests administered between April 2004 and June 2005) were collected for 7,208 adult ESL students in Massachusetts and Illinois. Program staff also contributed attendance data for the same students during this time period. BEST Plus has been used in Illinois since 2003 (approximately 56,000 tests had been administered as of mid-2007) and in Massachusetts since 2004 (approximately 86,000 tests had been administered as of mid-2007). A total of 6,599 examinees with complete data for analysis remained in the data set. The pretest NRS educational functioning levels of these examinees are shown in Table 3. (Percentages in this report are rounded up to the nearest whole number, so totals may be slightly higher or lower than 100%.)

Table 3. Distribution of Participants in NRS Levels Based on BEST Plus Pretest
Initial NRS Level	Number	Percentage
Beginning ESL Literacy	3,209	49%
Low Beginning ESL	519	8%
High Beginning ESL	709	11%
Low Intermediate ESL	989	15%
High Intermediate ESL	718	11%
Advanced ESL	455	7%
Total	6,599	100%

At the time that these data were collected, the original (1999-2005) NRS ESL educational functioning level tables were in use, comprising Beginning ESL Literacy, Beginning ESL, Low Intermediate ESL, High Intermediate ESL, Low Advanced ESL, and High Advanced ESL. However, these data were analyzed using the current NRS ESL levels (revised as of July 2006). For example, the old Beginning ESL was split into Low Beginning ESL and High Beginning ESL and the old High Advanced ESL was removed. Under the revised NRS level tables, the largest number of students pretested using BEST Plus were placed in the NRS Beginning ESL Literacy level (49%). The fewest number of students were placed in the NRS Advanced ESL level (7%). These numbers are similar to the overall breakdown of levels of all students enrolled in federally funded adult ESL programs in the United States in 2003-2004, the latest year for which these data are available (U.S. Department of Education, Office of Vocational and Adult Education, 2006).

Table 4 shows the number and percentage of students in the study who made level gain. The greatest rate of level gain, regardless of number of instructional hours, occurred in Low Beginning (78%) and High Beginning (77%) ESL levels, with Beginning ESL Literacy (54%) and Advanced ESL (55%) making the lowest rates of level gain. Overall, 60% of the students made one or more level gains.

Table 4. Number of Students Attaining NRS Educational Level Gain by NRS Level
Initial NRS Level	Number Making Level Gain	Percentage Making Level Gain
Beginning ESL Literacy n=3,209	1,720	54%
Low Beginning ESL n=519	407	78%
High Beginning ESL n=709	543	77%
Low Intermediate ESL n=989	614	62%
High Intermediate ESL n=718	408	57%
Advanced ESL n=455	252	55%
Total n=6,599	3,944	60%

Question 1: What Is the Relationship Between Instructional Hours and Educational Level Gain on BEST Plus?

BEST Plus was administered as a posttest to the 6,599 students after they received instruction ranging from 2 to 512 instructional hours. Approximately 38% of students in the 140 or more hours group received more than 200 instructional hours. For these data sets, instructional hours is de fined as the number of hours that students actually attended class, not the total number of hours offered in the program. The percentages of all examinees (regardless of NRS educational functioning level) who attained at least one level gain between the pretest and posttest are delineated by number of instructional hours in Table 5. Students with 140 or more instructional hours had the highest percentage of educational level gain on BEST Plus (70%), and students with fewer than 60 instructional hours had the lowest percentage of level gain (53%).

Table 5. Number of Students Attaining NRS Educational Level Gain by Instructional Hours
Instructional Hours	Attained Level Gain
Below 60 hours n=2,066	n=1,101 53%
60 to 79 hours n=1,194	n=660 55%
80 to 99 hours n=1,015	n=611 60%
100 to 119 hours n=630	n=391 62%
120 to 139 hours n=482	n=331 69%
140 hours or more n=1,212	n=850 70 %
Total n=6,599	n=3,944 60%

 

Table 6 breaks down the rates of level gain according to NRS educational functioning level and instructional hours. There is a general trend toward greater level gain with more instructional hours across all NRS levels. For each of the NRS educational functioning levels examined, a greater percentage of students who received 140 or more instructional hours made level gain compared to those students who received fewer than 60 instructional hours. However, the percentage of level gain is not always consistent with increasing numbers of instructional hours. These inconsistencies cannot be explained given the purely descriptive data that were analyzed.

Table 6. NRS Level Gain Related to Instructional Hours by NRS ESL Educational Functioning Level
Instr. Hours	Beginning ESL Literacy n=1,720	Low Beg. ESL n=407	High Beg. ESL n=543	Low Inter. ESL n=614	High Inter. ESL n=408	Advanced ESL n=252
Below 60 Hours	n=536 46%	n=112 75%	n=160 72%	n=146 60%	n=96 54%	n=51 50%
60 to 79 Hours	n=284 52%	n=86 79%	n=85 69%	n=93 51%	n=68 49%	n=44 48%
80 to 99 Hours	n=247 56%	n=57 72%	n=95 79%	n=105 60%	n=74 58%	n=33 45%
100 to 119 Hours	n=159 54%	n=34 81%	n=52 81%	n=75 68%	n=42 58%	n=29 64%
120 to 139 Hours	n=120 62%	n=37 86%	n=52 87%	n=49 66%	n=43 64%	n=30 67%
140 or More Hours	n=374 67%	n=81 84%	n=99 83%	n=146 71%	n=85 62%	n=65 66%

 

Summary of Results: Instructional Hours

Across NRS educational functioning levels, the general trend (with a few exceptions at some levels) is that the greater the number of instructional hours, the higher the percentage of students who made level gain. This is particularly true of students who pretested at the Beginning ESL Literacy level (21% difference between the fewest number and the greatest number of instructional hours) and the Advanced ESL level (16% difference). A high percentage of students in the Low Beginning and High Beginning levels made level gain, regardless of the number of instructional hours.

Question 2: What Is the Relationship Between Intensity of Instruction and Educational Level Gain on BEST Plus?

Student pretest and posttest performances on BEST Plus were also compared to intensity of instruction (i.e., how often students attend class). Measuring intensity of instruction provides different information about student attendance than measuring instructional hours. For this study, three levels of intensity were defined, based on a ratio of total hours of instruction divided by days between pretest and posttest:

• Low intensity (ratio less then .50); for example, 100 hours of instruction over 250 days
  
  
• Mid intensity (ratio between .50 and .99); for example, 100 hours of instruction over 150 days
  
  
• High intensity (ratio of 1.00 or above); for example, 100 hours of instruction over 75 days
  
  

Table 7 illustrates a general trend toward greater rates of NRS level gain for students with high intensity instruction (66% attained level gain) than for those with low intensity instruction (56% attained level gain).

Table 7. Overall NRS Level Gain by Intensity of Instruction
Intensity Level	Attained Level Gain
Low Intensity n=2,049	n=1,142 56%
Mid Intensity n=3,733	n=2,265 61 %
High Intensity n=817	n=537 66%
Total n=6,599	n=3,944 60%

Table 8 shows the rates of level gain according to NRS levels and intensity of instruction. Although all NRS levels demonstrated increased level gain with increased intensity of instruction, Table 8 indicates that intensity of instruction had the greatest impact on students at the Advanced ESL level (19% difference in rates of level gain between low intensity and high intensity groups).

Table 8. NRS Level Gain by Intensity of Instruction
Intensity Level	Beginning ESL Literacy n=1,720	Low Beg. ESL n=407	High Beg. ESL n=543	Low Inter. ESL n=614	High Inter. ESL n=408	Advanced ESL n=252
Low Intensity (<0.50)	n=548 50%	n=116 77%	n=165 75%	n=140 55%	n=121 57%	n=52 49%
Mid Intensity (0.50-0.99)	n=935 55%	n=247 78%	n=321 77%	n=386 64%	n=236 56%	n=140 54%
High Intensity (>1.00)	n=237 60%	n=44 83%	n=57 83%	n=88 68%	n=51 61%	n=60 68%

Summary of Results: Intensity of Instruction

Overall, there was a tendency for a higher percentage of students in more intensive instruction to achieve level gain: 66% of students in high intensity instruction made a level gain compared with 56% of students in low intensity instruction. This tendency is strongest in the Advanced ESL, Low Intermediate, and Beginning ESL Literacy levels, where the differences in level gain between low and high intensity range from 10% to 19%, as opposed to 4% to 8% for the other levels.

Conclusion

Overall, NRS level gain was achieved for 60% of the 6,599 students whose pretest and posttest scores were included in the analyses. Of particular note, 78% of the Low Beginning ESL students and 77% of the High Beginning ESL students in the study achieved NRS level gain regardless of the number of instructional hours. These rates contrast with lower rates for students at the other NRS levels. They are also higher than the overall average of 36% level gain reported for all students in federally funded programs in 2003-2004 (U.S. Department of Education, Office of Vocational and Adult Education, 2006). Intensity of instruction also affected NRS level gain, with the greatest impact found in students at the Advanced ESL level.

It is not possible to pinpoint the reasons for the differences among levels based solely on the data analyzed in this study. The differences may indicate that students are acquiring more oral English language at a faster rate at Low Beginning ESL and High Beginning ESL levels than at other levels. This increased rate of acquisition may be related to a variety of factors, such as the curriculum and materials used at these levels, or beginning students’ levels of motivation. In accordance with lower level students’ limited range of knowledge and skills in English, beginning ESL instruction usually focuses on a much more limited set of information (e.g., personal information, numbers, household vocabulary, simple sentences). As adult English language learners’ oral proficiency develops, a much greater range of skills must be demonstrated (e.g., supporting opinions, comparing and contrasting, elaborating on experiences) that may require a proportionally greater amount of time in order to show progress.

Second language acquisition theories may also be considered. (See Doughty & Long, 2005, for an overview of related hypotheses.) For example, the silent period hypothesis proposes that beginning language learners go through a period in which their receptive skills are stronger than their productive skills. Another theory involves language fossilization, in which language learners reach a plateau, typically at more advanced stages of their language acquisition, and have difficulty making further progress.

Additional research is needed to identify (1) reasons for the differences in level gain found in this study and (2) learner and programmatic factors, beyond number of instructional hours and intensity of instruction, that may influence NRS level gain. The results of the statistical analyses show that the number of instructional hours had a greater effect on overall group NRS level gain, as measured by BEST Plus, than instructional intensity. It should be noted that this study describes clean data related to 6,599 participants. However, these participants were not randomly selected; personal, environmental, and programmatic variables were not controlled; and there were not equal numbers of students from each NRS educational functioning level. Because this report documents a descriptive study rather than a controlled correlational study, these preliminary results cannot be interpreted as cause-and-effect relationships among the variables. Rather, these preliminary results can serve as the basis for examining the NRS level gain of the 6,599 adult ESL students whose data were included in the analyses. In order to make stronger correlational claims between instructional hours and NRS level gain as measured by BEST Plus, more complete data about randomly selected examinees and their programs need to be collected and analyzed in a more experimental study.

Recommendations for Adult ESL Programs The Center for Applied Linguistics continues to recommend 80 to 100 instructional hours between pretesting and posttesting on BEST Plus. The findings from this descriptive study regarding instructional hours support and strengthen this recommendation. This should be taken into consideration, along with other factors affecting second language acquisition and adult English language learners, such as those described above, when determining state and local assessment policy for pre and posttesting. To increase the likelihood of NRS level gain by adult English language learners, adult ESL programs should consider doing the following: Conduct a needs assessment of local learners to determine the best days, times, and locations to meet the scheduling needs of the adult ESL population in the community. Offer a variety of available instructional hours at various sites based on the needs assessment. Identify ways to increase student attendance and persistence specific to the local area. Provide information to students, instructors, funders, and community stakeholders about the number of instructional hours needed to show educational level gain. Inform new students at orientation and during intake processes of the importance of class attendance and participation in pretesting and posttesting; make sure students understand how pretesting and posttesting with a standardized assessment relate to the program's funding sources and ability to track student progress. Train instructors to make the most of instructional time by conducting student needs assessments, developing effective lesson plans, selecting appropriate materials, and evaluating students’ progress on an ongoing basis. Teach students how to apply and reinforce the English language classroom instruction they receive to their interactions outside of the classroom, as doing so will enhance their English language development.

References

Coady, J., & Huckin, T. (Eds.). (1997). Second language vocabulary acquisition: A rationale for pedagogy. Cambridge, UK: Cambridge University Press.

Dörnyei, Z. (2002). Teaching and researching motivation. Essex, UK: Pearson Education.

Doughty, C., & Long, M. (Eds.). (2005). The handbook of second language acquisition. Malden, MA: Blackwell.

Ellis, R. (2000). Task-based research and language pedagogy. Language Teaching Research, 4, 193-220.

Long, M. H. (1996). The role of the linguistic environment in second language acquisition. In W. C. Ritchie & T. K. Bahtia (Eds.), Handbook of second language acquisition (pp. 413-68). New York: Academic Press.

Moss, D., & Ross-Feldman, L. (2003). Second language acquisition in adults: From research to practice. Washington, DC: Center for Applied Linguistics. Retrieved August 16, 2007, from www.cal.org/CAELA/esl_resources/digests/SLA.html

National Center for ESL Literacy Education. (2003). Adult English language instruction in the 21st century. Washington, DC: Center for Applied Linguistics.

National Reporting System. (2006, May). State assessment policy guidance. Retrieved August 16, 2007, from www.nrsweb.org

Teachers of English to Speakers of Other Languages. (2003). Standards for adult education ESL programs. Alexandria, VA: Author.

U.S. Department of Education, Office of Vocational and Adult Education. (2006). Adult Education and Family Literacy Act, Report to Congress on state performance, 2003-2004. Retrieved August 16, 2007, from www.ed.gov/about/offices/list/ovae/index.html

Acknowledgements

The data for this report were collected and analyzed by members of CAL’s Language Testing Division, including Dr. Dorry M. Kenyon, director, and Willow Ryu, quantitative research specialist. The report was reviewed by members of CAL’s Adult ESL Assessments team: Frank Finamore, project manager; Michelle Ueland, client relations manager; and Daniel Lieberson, products manager. Dr. Joy Kreeft Peyton, director, Miriam Burt, associate director, and Sharon McKay, adult ESL specialist with the Center for Adult English Language Acquisition (CAELA), reviewed the report as well. Special thanks go to Donna Corneiller and Jane Schwertfedger of the Massachusetts Department of Education for their review of this document. Finally, our gratitude goes out to the Illinois Community College Board and the Massachusetts Department of Education for contributing the pretest and posttest data for the adult English language learners who were included in the study.

Learn more about CAL's adult ESL assessments: BEST Plus and BEST Literacy.

December 2007