Download a PDF of this digest.

Oral proficiency assessment allows test takers to show how well they can communicate orally in a language. There are a number of approaches to oral proficiency assessment, including direct and semi-direct means. Direct approachesinclude assessments conducted by a live interviewer who elicits language from the test taker via a face-to-face or telephonic oral proficiency interview (OPI). The American Council on the Teaching of Foreign Languages (ACTFL) has conducted thousands of oral proficiency interviews to assess the oral proficiency of students and professionals in the United States over the past 25 years. But face-to-face or telephonic testing of oral proficiency can be expensive and time consuming, which often makes semi-direct testing more attractive to test users.

Semi-direct approaches refer to testing methods that rely on something other than a live interviewer—such as a tape recorder, computer program, test booklet, or some combination of these—to elicit language from the test taker. It is important to select an approach that is appropriate for each testing situation.

To date, most approaches to oral proficiency assessment require a trained rater to listen to and rate the examinee's performance. For direct assessments, the interviewer usually also serves as the rater; for semi-direct assessments, the performance is rated at a later time by a trained rater. Providing reliable training for interviewers and raters comprises a great deal of the cost of oral proficiency testing.

Since 1986, the Center for Applied Linguistics (CAL) has investigated and initiated a number of approaches to oral proficiency assessment that rely on semi-direct measures and new technologies. This digest describes ways to use technology to improve test development and administration and to improve the training of raters. CAL projects are used to illustrate each of these areas.

Using Technology to Improve Testing

Past approaches to oral proficiency assessment relied on tape recorders to elicit language from examinees (Malone, 2003). As technology has progressed, new approaches have been developed that rely on various computer technologies, such as CD-ROMs, the World Wide Web, and computers' adaptive capabilities. This section outlines three computer-based assessment projects conducted by CAL: a computer-delivered oral proficiency assessment for postsecondary students, a research project on oral proficiency assessment, and a computer-assisted oral interview for adult learners.

The Computerized Oral Proficiency Instrument
The Computerized Oral Proficiency Instrument (COPI) is a computer-adaptive oral proficiency test for postsecondary students administered via CD-ROM. Its design stems from the ACTFL OPI and from CAL's Simulated Oral Proficiency Interview (SOPI), a tape-mediated assessment. From 1998 to 2000, CAL conducted a study to determine the feasibility of a computer-administered oral proficiency test (Kenyon & Malabonga, 2001). This study confirmed that the COPI could reliably elicit language from examinees and produce results comparable to those elicited from the ACTFL OPI and CAL SOPI (Kenyon & Malabonga, 2001). The COPI design capitalizes on the strengths of technology. Test takers are able to control a great deal of the test, including how much time they take to plan and give their responses.

Based on the feasibility study, CAL received funding to operationalize the COPI so that it can be used by language programs. COPI tasks have been evaluated and revised based on feedback from the feasibility study. During 2007, the new version of the COPI, available in Arabic and Spanish, will be field-tested with examinees in both languages. The field test will provide information on the COPI’s reliability. Following the field test, CAL will develop rating procedures for the COPI via distance-learning technologies.

The COPI reflects the need for versatile applications of technology to oral proficiency testing. The CD-ROM technology will enable programs to assess large numbers of students simultaneously in a cost-efficient manner, and digitized responses can be easily transferred to raters. Rating time for the COPI is three times faster than for the SOPI (Malabonga, Kenyon, & Carpenter, 2005). This will provide a cost-efficient scoring program, as raters are not needed to administer the exam individually and can score the responses at a later date.

The Computer-Assisted Screening Tool
The Computer Assisted Screening Tool (CAST) Framework investigated the characteristics of oral proficiency tasks that best elicit oral proficiency samples from test takers. This project involved a consortium of language testers from ACTFL, Brigham Young University, the Defense Language Institute, and San Diego State University. By examining more than 50 examinees’ responses to the Arabic and Spanish OPI, SOPI, and COPI, researchers at CAL determined the characteristics of oral proficiency tasks that most efficiently elicit responses from test takers. The results have been compiled into a framework, or description, of the characteristics of optimal oral proficiency assessment tasks.

During the CAST project, ACTFL-certified OPI testers reviewed and rated hundreds of samples from OPIs, SOPIs, and COPIs in Arabic and Spanish. Raters evaluated the tasks and responses to determine whether each task elicited the appropriate level of speech. CAL staff analyzed the data and wrote item specifications for task development for level-specific tests at each of the four ACTFL levels (Novice, Intermediate, Advanced, and Superior). Based on these task specifications, partners at San Diego State University developed Web-delivered tests of Arabic and Spanish. These tests were piloted during 2006.

BEST Plus
BEST Plus is a computer-adaptive assessment on CD that is administered as a face-to-face oral interview. The computer prompts the test administrator to ask specific questions that appear on the computer screen. After listening to the examinee’s answer, the administrator rates the response according to a rubric and inputs the rating into the computer. The computer then uses that information to choose the next test question. This computer-adaptive test combines a face-to-face approach to testing with the ease of computerized administration and scoring. This means that the test questions are generated based on the test administrators’ ratings as the interview progresses.

Using Technology to Train Raters of Oral Proficiency Tests

Because rater reliability, or a rater’s ability to rate consistently and in the same way as other raters, is crucial to the effectiveness of a test, appropriate training for OPI interviewers and SOPI raters is essential. Historically, both OPI interviewers and SOPI raters have been trained through a face-to-face instructional model, which allows prospective interviewers and raters to participate in training with an experienced instructor and colleagues. This model has proven both effective and expensive.

In the 1990s, to reduce the time and cost involved in rater training, CAL developed a set of self-instructional materials for SOPI raters. The materials included cassette tapes with pre-rated speech samples and an accompanying training manual to explain the rating system. While research showed this to be a reliable way of training raters (Kenyon, 1997), CD-ROM and Internet technology have provided additional approaches to training raters through more interactive means. A CD-ROM approach and a Web-based distance learning approach are described below.

The Multimedia Rater Training Program (MRTP)
The MRTP is a software program designed to teach language professionals to rate oral language proficiency. This computer-assisted professional development program was modeled after live rater training workshops and CAL’s Self-Instructional Rater Training Kits. The MRTP provides a hands-on introduction to oral proficiency assessment and teaches rating skills to language professionals conveniently and effectively using a CD.

The program trains language professionals to understand and apply the ACTFL Proficiency Guidelines – Speaking: Revised 1999 (American Council on the Teaching of Foreign Languages, 1999). The Guidelines are presented in modules that encourage learning through interactive tasks. Furthermore, the Guidelines are made concrete through examples of authentic speech samples at the various levels of proficiency.

Through the MRTP, language professionals learn to rate the SOPI, a tape-mediated speaking test based on ACTFL’s face-to-face oral proficiency interview. On SOPIs, students record their responses to 15 tasks on either a tape recorder or a computer.

The MRTP is currently available in Spanish. French and German versions are in development.

Assessment Training Online (ATOL)
ATOL courses use distance learning to teach participants to rate oral proficiency. The first online course, the Core Course, includes an introduction to the ACTFL Proficiency Guidelines-Speaking and provides practice applying this scale to rating samples. This is accomplished through online readings, weekly discussion boards, chat sessions, and assignments submitted electronically. After participants complete the Core Course, they are qualified to take a language-specific rater training course in either Arabic or Spanish. These courses train professionals to apply the rating scale to their particular language through online readings, discussion boards, chat sessions, and electronic assignments. Participants listen to and rate speech samples in the target language and complete quizzes to monitor their rating performance.

The Core Course and the Spanish and Arabic courses have been piloted with language professionals. ATOL courses provide low-cost alternatives to face-to-face live rater training. Course participants were surveyed about their experience, and the majority found distance learning to be an effective mode of delivery that successfully presented written, audio, and graphic course materials. The ATOL courses provide a cost-effective method to train raters in various locations simultaneously.

Conclusion

As new technologies become available and as current technologies become more accessible and affordable, assessments have moved beyond tape-mediated and face-to-face or telephonic tests to embrace computer technologies. Most importantly, these technologies have been integrated not only into test development but also into rater training, which is often costly to conduct in face-to-face situations. Future tests may include online, video-based, and face-to-face assessments with high levels of validity and reliability.

The projects outlined in this digest highlight four aspects of developing oral proficiency tests:

• The CAST Framework has investigated and analyzed the characteristics of successful oral proficiency tasks.
• The COPI project will review, improve, and operationalize a test.
• The ATOL uses a Web-based distance learning approach to train raters.
• The MRTP relies upon an interactive CD-ROM to train users to rate oral language.

When selecting a test, it is important for test users to determine which approach—direct or semi-direct, tape-mediated or computerized, CD-ROM-based or Web-delivered—is appropriate for their particular context.

References

American Council on the Teaching of Foreign Languages. (1999). ACTFL proficiency guidelines—speaking: Revised 1999. Retrieved June 4, 2007, from http://www.actfl.org/files/public/Guidelinesspeak.pdf

Kenyon, D. M. (1997). Further research on the efficacy of rater self-training. In A. Huhta, V. Kohonen, L. Kurki-Suonio, & S. Luoma (Eds.), Current developments and alternatives in language assessment: Proceedings of LTRC 96 (pp. 257-273). Jyväskylä, Finland: University of Jyväskylä.

Kenyon, D., & Malabonga, V. (2001). Comparing examinee attitudes toward computer-assisted and other oral proficiency assessments. Language Learning & Technology, 5(2), 60-83.

Malabonga, V., Kenyon, D., & Carpenter, H. (2005). Self-assessment, preparation and response time on a computerized oral proficiency test. Language Testing, 22(1), 59-92.

Malone, M. E. (2003). Research on the oral proficiency interview: Analysis, synthesis, and future directions. Foreign Language Annals, 36(4), 491-497.

October 2007