Promoting education and achievement of adults learning English
Using Adult ESL Content Standards
Kirsten Schaetzel and Sarah Young
Center for Adult English Language Acquisition, Center for Applied Linguistics
Download a PDF of this brief (201 KB)
Background on Adult Learners
Adult education programs serve learners who are native English speakers and those whose first, or native, language is not English. Native English speakers attend adult basic education (ABE) classes to learn the skills needed to earn high school equivalency certificates or to achieve other goals related to job, family, or further education. English language learners attend English as a second language (ESL) or ABE classes to improve their oral and written skills in English and to achieve goals similar to those of native English speakers.
Audience for This Brief
This brief is written for adult ESL teachers and program administrators, as well as educational researchers, policymakers, and stakeholders who work with adult English language students in ESL classes or in mixed ABE classes (with native English speakers and English language students).
Content standards are defined as what learners should know and be able to do in a certain subject or practical domain (American Institutes for Research and U. S. Department of Education, Office of Vocational and Adult Education, 2005). A previous CAELA brief, Understanding Adult ESL Content Standards (Young & Smith, 2006), describes what content standards are, which adult ESL content standards are in use, and what research says about content standards implementation (www.cal.org/caela/esl_resources/briefs/contentstandards.html). This brief, Using Adult ESL Content Standards, begins with historical information about content standards and then describes the processes that adult ESL teachers and program administrators can follow to successfully incorporate standards into lesson planning, classroom activities, performance assessment, and professional development.
Background on content standards
The standards movement in adult ESL education grew out of the standards movement in elementary and secondary education (Marzano, 1998; Stites, 1999). The use of standards in adult English language programs reflects two important changes over the past fifty years, one in education in general and one in English language teaching. First, for many years in education, teacher accountability was determined by the amount of information a teacher taught in class, specifically how much of the course text he or she was able to cover. Newer measures of accountability focus on what students learn (Daggett, 2000). Standards reflect this focus on student learning and are one way of determining what students need to learn and do learn.
A second change pertinent to the use of standards in adult ESL education occurred in the 1980s, when the field of English language teaching moved from grammar-based to communicative and content-based methodologies (Brinton, Snow, & Wesche, 1989; Brown, 2000; Gersten & Hudelson, 2000; Savignon, 1983). Communicative and content-based methodologies do not dispense with specific grammar teaching but rather focus on teaching grammar in communicative, content-related contexts. Communicative methodology teaches language through real-life communication, with the teacher setting the stage for language to be used in both formal and informal situations (Savignon, 1983). Content-based methodology gives a context for language learning so that students can apply the language learned in the classroom to real life contexts (Brinton, Snow, & Wesche, 1989). With the move from grammar-based to communicative and content-based methodologies, adult ESL programs moved from grammar curricula to communicative, content-based curricula.
Some programs abandoned the grammar curriculum but did not replace it with a curriculum based on communicative, content-based methodologies. As a result, some new and even experienced teachers were not certain what to teach and how to implement communicative, content-based instruction. They selected textbooks with a focus on communicative, content-based instruction, but the textbook material became the entire course (Brinton, Snow, & Wesche, 1989).
Content standards have helped to remedy this situation. When teachers and students have clear standards, competencies, benchmarks, and curricula to guide their teaching and learning, the result may be systematic learning in their classes. Weiss, Pasley, Smith, Banilower, and Heck (2003) stress that “teachers need a coherent set of messages and clear goals to guide their instructional choices” (p.34). Standards give teachers a coherent set of messages that guide instruction and help them identify what their students should know and the extent to which they know it (Spohn & Zafft, 2006). With standards to describe what to teach at each level, teachers may be better able to use communicative language learning and content-based instruction in an efficient and effective manner.
What teachers need to consider
In their review of the influence of standards on K-12 teaching and learning, Lauer et al. (2005) conclude that “any impacts of state standards and statewide initiatives on teacher instruction are mediated by the development of curricula, materials, and instructional guidelines aligned with those particular standards and supported by professional development” (p.46). In standards-based education, a curriculum is used jointly with content standards that are usually organized by language skill (speaking, listening, reading, writing) and student proficiency level (literacy, beginning, intermediate, advanced). Adult ESL curricula may vary from program to program within a state, but the state ESL content standards provide a guiding framework for what students should know and be able to do as a result of instruction. Some curricula may be based on a textbook series, while others may be designed to address life skills (such as health, consumerism, or work) or program type (such as workplace, family literacy, or academic transitions). In the development of state content standards, existing curricula are often consulted to inform the process.
Based on summaries of research and practice on content standards (Carr & Harris, 2001; Marzano, 1998; Samway, 2000), the following questions have been developed for teachers to consider when incorporating standards into their lesson planning, classroom activities, and assessment. Figure 1 (Standards-Based Lesson Planning Process) provides an outline of the components included in standards-based instruction.
What are students’ needs and goals?
In order to know students’ needs and goals, it is important to do a needs assessment at the beginning of the class (Auerbach, 1994; Weddel & Van Duzer, 1997). Teachers who know students’ needs and goals for learning may provide an outline for the course aligned with students’ needs. Teachers may not wish to use the exact terms of standards with students, but they can help students understand the goals of relevant standards by asking them to rewrite a standard (or overall concept) in their own words and provide examples of what it means or how it relates to their own needs and goals. Students may also wish to keep a checklist of the progress indicators or benchmarks that they are working toward in a particular unit. When students look back over the standards that they have mastered, they can see visible progress toward meeting their goals (Carr & Harris, 2001).
For example, a listening standard for Intermediate ESL is, “Follow simple two-step directions and instructions” (Maryland State Department of Education, 2003). To teach a lesson based on this standard aligned with students’ needs and goals, a teacher should first determine, by asking the students, in which areas of their lives they need to follow simple directions (e.g. at work, in the bank, at their children’s school, at their children’s after-school program). The answers provide a context for a lesson focused on following simple two-step directions.
What does the standard emphasize and focus on?
In planning a lesson based on standards and its related benchmarks, a teacher should first analyze the standard to determine the teaching focus (Carr & Harris, 2001; Coffey, 2006). Adapting Coffey’s, and Carr and Harris’ analysis, two questions a teacher can ask to analyze a standard are
- What skill(s) does the standard cover? (e.g. reading, writing, listening, speaking)
- What are indicators of student progress in meeting the standard (often called benchmarks)?
For example, in the Reading Strand of the Massachusetts ABE English for Speakers of Other Languages Curriculum Framework, Standard 1 is, “English language learners will read and comprehend a variety of English texts for various purposes” (Massachusetts Department of Education, 2005). The benchmarks for this standard describe what students should be able to do when completing each class level. When the above questions are asked about this standard, the standard and its benchmarks can be analyzed as follows:
- What skill(s) does the standard cover? It covers reading skills, specifically reading comprehension.
- What are indicators of student progress in meeting the standard? Teachers can look at the benchmarks listed for their level in order to determine which benchmark will be the focus for a particular lesson. For example, for the Low Beginning level, the benchmarks are that students can 1) understand short, simple paragraphs . . . on a familiar topic; 2) scan and extract relevant information (e.g. forms, labels, maps, etc.); and 3) read and follow simple, familiar, one-step written directions.
A teacher can focus a lesson by analyzing the standard to determine the skills that will be worked on, the appropriate benchmarks of progress, and the features needed in the class materials.
What are the components of a standards-based lesson?
Lessons that form part of a standards-based curriculum unit begin with well-articulated objectives that link to the chosen content standards and benchmarks, as determined by a student needs assessment and guided by the standards framework appropriate for the given level. In order to make the lesson objectives achievable and measurable, each lesson should focus on only one or two broad standards (e.g., “Organize and relay information to effectively serve the purpose, context, and listener;” Equipped for the Future, 2000). Teachers may choose to track or color code the standards covered within a specific instructional period in their lesson plan or curriculum records, to quickly determine if a balance of language skills is being achieved.
Each content standard is accompanied by progress indicators or benchmarks arranged by level (see Young & Smith, 2006) to provide clear examples of the skills needed to demonstrate student progress toward meeting the standard (American Institutes for Research & U.S. Department of Education, Office of Vocational and Adult Education, 2005). The curriculum content (e.g., life skills, family literacy) is related to the standards through these indicators and benchmarks. Using the progress indicators or benchmarks, core ideas and concepts to be covered are identified, including ways to have students use progressively higher level thinking skills (Teachers of English to Speakers of Other Languages, 2001).
The lesson or learning objective is developed from the core ideas and concepts to be covered in a lesson, which are informed by both the curriculum unit and the standard’s progress indicators or benchmarks (see Figure 1: Standards-Based Lesson Planning Process). An effective lesson objective demonstrates the following:
- The specific context in which the activity will be carried out;
- one or two language skills that will be covered (listening, speaking, reading, writing);
- one communicative task or purpose; and
- a way that mastery or completion of the specified skill will be evaluated
(Center for Applied Linguistics, in press).
Examples of objectives that adhere to these guidelines are, “Make a grocery list of healthy food choices,” “Read a narrative paragraph about Cinco de Mayo and answer comprehension questions,” and “Take and leave simple phone messages at home” (Adelson-Goldstein & Owensby, 2005).
The resulting structure of the lesson plan and activities should follow the benchmarks and objectives to gradually build students’ skills and knowledge through practice and application, in preparation for a final evaluation of their ability to complete the task or demonstrate the required knowledge (Gillespie, 2002; National Center for Family Literacy & Center for Applied Linguistics, 2004). For example, the Splendid ESOL Web site (http://cc.pima.edu/~slundquist/index.htm ) links Arizona’s content standards with language functions and activities to help students meet those standards. At Level 1 (Beginning ESL), one of the functions is “Cautions and warns.” Based on an assessment of student needs, one classroom activity uses a picture dictionary to show students what they need to be warned about when living in the desert of Arizona (e.g., snakes, cactus, lack of water). An accompanying computer activity takes students to the University of Arizona’s College of Pharmacy Web site and shows them how to find information about venomous snakes and dangerous animals. They are given the task of looking for symptoms when a poisonous snake bites someone and first aid is needed. The objectives of this lesson focus on a context that is important and relevant to students’ lives in the desert of Arizona, the focus is on speaking and listening skills, and there is a clear communicative purpose.
What are standards-based materials?
A well-written standard will allow teachers the flexibility to use a variety of materials to illustrate and evaluate the content that students are learning. For example, Figure 2 lists four computer lab and classroom activities that align with Arizona’s English Language Acquisition for Adults Standards and language functions for students in Beginning ESL level 1. Each activity represents a type of print or online materials that may be authentic or based on authentic information.
FBI’s Most Wanted List
Standard: The adult English learner comprehends and communicates in written and spoken English for a variety of purposes and audiences.
Level: 1 (Beginning ESL)
Figure 2. Using Internet-Based Materials
A wide variety of materials will keep teachers and students engaged and informed and will address different learning styles (Christison, 2005; Gardner, 1993). Some adult ESL textbooks have been correlated to state and national content standards, such as EFF and CASAS, making it easier for teachers to find appropriate textbook materials based on a given standard (see, e.g., Cambridge University Press, Interchange Books 1-4; Oxford University Press, Grammar Sense Books 1-3). Teachers can build their own standards-based material collections by indicating or coding the standards and benchmarks addressed on worksheets and lesson plans, and then filing the materials appropriately.
The following should be considered when selecting or creating materials for a standards-based lesson or activity:
- Does it address the needs of the students?
- Does it contain the language features (e.g., language functions, grammatical structures, vocabulary) that the standard or benchmark requires?
- Does it match the oral English and literacy levels of the students?
- Is it engaging and interesting to the students?
- Does it require students to practice skills measured in the standard or benchmark?
- Can it be tailored or adapted for different student needs?
(For more information on selecting or creating materials for the language classroom, see Nunan & Lamb, 1996; Tomlinson, 2003.)
What are standards-based assessments?
Measurement of student progress on a particular standard develops from the progress indicators or benchmarks and subsequent lesson objectives. Before engaging students in the lesson, a teacher should have a clear idea of the intended outcome of an instructional activity – what students should know and be able to do (Wiggins & McTighe, 1998). Once the progress indicators or benchmarks are identified, the instructional activities and ongoing assessments should build and track the necessary skills to master the final assessment task, performance, product, or project. Ongoing (or formative) assessment tools are linked to progress indicators, are carefully integrated throughout a sequenced unit of instruction, and allow teachers to monitor students’ progress toward a content standard. Summative (or final) assessment tools at the end of a unit of instruction measure students’ performance on a collection of indicators related to the overarching standard. These may be used to inform instruction and also for accountability purposes through the National Reporting System. The combined use of formative and summative assessments provides benefits for students and teachers, as shown in Figure 3.
The combination of formative and summative assessment:
TESOL (2001, p. 7) outlines a standards-based instruction and assessment cycle that incorporates four steps, as shown in Figure 4.
Create a sequence of instructional activities, based on indicators or benchmarks found in the standards, that lead to the final standards-based outcomes
Collecting and recording information
Determine the formative and summative assessment tools to measure outcomes
Analyzing and interpreting information
Comparing current and previous performances to measure progress of individual students and the class as a whole
Reporting and decision making
Provide feedback to students on their mastery of the appropriate benchmarks and determine next steps, if more practice is needed
Figure 4. Standards-Based Instruction and Assessment Cycle
There are many ways that students can demonstrate progress in meeting standards. (A comprehensive list of general classroom assessment tools and measures can be found on the Equipped for the Future Web site at http://eff.cls.utk.edu/toolkit/support_ongoing_assessment.htm.) Because standards and related benchmarks and lesson objectives describe in a measurable way what students know and can do, performance-based assessments often allow students to demonstrate this much better than traditional types of tests (Ananda, 2000; Moon & Callahan, 2001). Performance-based assessments (such as role plays, demonstrations, written products, and exhibitions) relate back to the selected benchmarks. These types of assessments use rubrics or checklists to show how well students perform the task and to inform students of what constitutes an acceptable performance. Once students have an idea of how a lesson links to the standard and their own needs and goals, they may be able to help develop the components of a rubric or checklist to evaluate success on a particular performance-based task. These rubrics can then be completed not only by the teacher, but also by the students for self and peer assessments. Results may also indicate how well the students might complete a similar task outside of the classroom in their daily lives.
What program administrators need to consider
After the extensive process that state adult ESL content standards development teams undergo to create, pilot, and revise standards, program administrators are often faced with the challenge of getting the standards off the shelf and into the classrooms. Program administrators should seek the expertise of state education offices and content standards specialists to provide the necessary support for teachers in the following ways:
- Offer professional development for novice and experienced teachers when the standards are initially rolled out. Trainings should provide teachers with a rationale for the implementation of standards and expectations for how the use of standards will shape instructional design and student learning and outcomes. Because many content standards documents tend to be lengthy and detailed, teachers should receive many practical examples of what the standards and benchmarks mean for the classroom at each learner level as well as suggested materials, activities, and formative and summative assessments that are aligned with the standards. Professional development should be ongoing, with orientations provided to new teachers who join staff later on and refresher trainings for more experienced teachers.
- Provide opportunities for collaboration among teachers who teach not only similar but also adjacent levels. By working with colleagues who teach one level above or below that of their students and comparing how benchmarks evolve across levels within a particular standard (see Figure 5), teachers have a clearer picture of what students’ expected progress is from one level to the next and how they can better prepare their students for advancing. Working with staff developers, teachers can collaborate to design a system of assessments and performance-based tasks with established criteria for success that effectively connect student progress from one level to the next.
Adjacent Benchmarks for Reading Standard 1 of the Massachusetts ABE English for Speakers of Other Languages Curriculum Framework (2005)
Reading Strand, Standard 1: English language learners will read and comprehend a variety of English texts for various purposes.
Sample benchmark, Low Intermediate:
Sample benchmark, High Intermediate:
Sample benchmark, Advanced:
- Encourage teacher creativity in designing and implementing standards-based curriculum, lessons, materials, activities, and assessments. Standards and their related benchmarks are meant to guide teachers in designing instruction, but they should not be seen as prescriptive or limiting. Teachers should be aware of the variety of ways that they can use standards and measure progress.
Student needs assessments provide essential information about what and how students want to learn, and content standards give teachers a common framework for designing instruction and for describing how students are learning and progressing in their classes. Teachers can prepare students to work, live, and communicate outside of the ESL classroom by focusing on what students know and can do and by tailoring instruction and assessment to match students’ needs. The use of this standards-based education can help to professionalize the field of teaching adult English language learners. In light of this, OVAE is planning future technical assistance to support states in moving toward the goals of standards-based education for adults (See www.ed.gov/about/offices/list/ovae/pi/AdultEd/stndassess.html for more information).
As standards-based education becomes more common in adult ESL practice, additional research and resources are needed to develop and implement instruction and assessment that align to the standards. Much of the research on the success of standards implementation comes from the K-12 field. With the K-12 efforts to build on, and more research and resources developed that specifically address using standards with adult English language learners, the quality of the teaching and learning in adult ESL classes can be significantly improved. As Stites (1999, p. 7) points out, “Adult educators joined the standards fray rather late. In some ways, this is an advantage. As late-adopters we can benefit from the successes and failures of the K-12 efforts.”
The authors wish to thank the following adult ESL teachers and content standards specialists who shared their ideas, experiences, and suggestions for this brief:
Claudia Bianca-DeBay, Truckee Meadows Community College, Reno, NV
Susan Lundquist and Adele Youmans, Pima College, Tucson, AZ
Cathy McCargo and Mary Jane Tuckerman, Arlington Education and Employment Program, Arlington, VA
Andrea O’Brien, The Adult Learning Center, Lawrence, MA
Adelson-Goldstein, J., & Owensby, J. (2005). An objective approach to lesson planning. Los Angeles: ESL/CBET & Citizenship Programs, Division of Adult and Career Education, Los Angeles Unified School District.
American Institutes for Research and U.S. Department of Education, Office of Vocational and Adult Education. (2005). A process guide for establishing state adult education content standards. Washington, DC: U.S. Department of Education. Available at www.adultedcontentstandards.ed.gov/howto.asp
Ananda, S. (2000). Equipped for the Future assessment report: How instructors can
support adult learners through performance-based assessment (EX 0110P). Washington, DC: National Institute for Literacy.
Auerbach, E. (1994). Making meaning, making change: Participatory curriculum development for adult ESL literacy. Washington, DC and McHenry, IL: Center for Applied Linguistics and Delta Systems. (ERIC No. ED 356 688)
Brinton, D., Snow, M., & Wesche, M. (1989). Content-based second language instruction. Boston: Heinle and Heinle
Brown, H.D. (2000). Principles of language learning and teaching. (4th edition). White Plains, NY: Pearson.
Cambridge University Press, Interchange Books 1-4. Available at www.cambridge.org/us/esl/correlations.htm
Carr, J., & Harris, D. (2001). Succeeding with standards: Linking curriculum, assessment, and action planning. Alexandria, VA: Association for Supervision and Curriculum Development.
Center for Applied Linguistics. (in press). CAELA guide for adult ESL trainers. Washington, DC: Center for Adult English Language Acquisition.
Christison, M. (2005). Multiple intelligences and language learning: A guidebook of theory, activities, inventories, and resources. Burlingame, CA: Alta Books.
Coffey, N. (2006). Women’s literature and the ELA framework. Field Notes, 15(4), 1, 3-4, 10.
Daggett, W. (2000). Moving from standards to instructional practice. NASSP Bulletin, 84(620), 66-72.
Equipped for the Future. (2000). Speak so others can understand. Washington, DC: National Institute for Literacy. Retrieved December 27, 2006, from http://eff.cls.utk.edu/PDF/03Speak.pdf
Gardner, H. (1993). Frames of mind: The theory of multiple intelligence. (10th edition) NY: Basic Books.
Gersten, B., & Hudelson, S. (2000). Developments in second language acquisition research and theory: From structuralism to social participation. In M. Snow (Ed.), Implementing the ESL standards for Pre-K-12 students through teacher education (pp. 75-102). Alexandria, VA: Teachers of English to Speakers of Other Languages.
Gillespie, M. (2002, October). EFF research principle: A contextualized approach to
curriculum and instruction (EFF Research to Practice Note 3). Retrieved December 27, 2006, from http://eff.cls.utk.edu/PDF/03research-practice.pdf
Lauer, P., Snow, D., Martin-Glenn, M., Van Buher., R. Stoutemyer, K., & Snow-Renner, R. (2005). The influence of standards on K-12 teaching and student learning: A research synthesis. Aurora, CO: Mid-continent Research for Education and Learning. Retrieved December 27, 2006 from www.mcrel.org/PDF/Synthesis/5052_RSInfluenceofStandards.pdf
Maryland State Department of Education. (2003). Maryland content standards for adult ESL/ESOL. Baltimore, MD: Author.
Marzano, R. (1998). A new year of school reform: Going where the research takes us. Aurora, Co: Mid-Continent Research for Education and Learning. Retrieved December 27, 2006, from www.mcrel.org/topics/productDetail.asp?productID=81
Massachusetts Department of Education. (2005). Massachusetts adult basic education curriculum framework for English for speakers of other languages. Retrieved December 27, 2006, from www.doe.mass.edu/acls/frameworks/esol.pdf
Moon, T., & Callahan, C. (2001). Classroom performance assessment: What should it
look like in a standards-based classroom? NASSP Bulletin, 85, 48-58.
National Center for Family Literacy & Center for Applied Linguistics. (2004). Practitioner toolkit: Working with adult English language learners. Louisville, KY and Washington, DC: Authors. Available at www.cal.org/caela/tools/program_development/CombinedFilesl.pdf
Nunan, D., & Lamb, C. (1996). The self-directed teacher. NY: Cambridge University Press.
Oxford University Press, Grammar Sense Books 1-3. www.us.oup.com/us/corporate/publishingprograms/esl/correlations/?view=usa
Samway, K. (Ed.). (2000). Integrating the ESL standards into classroom practice: Grades 3-5. Alexandria, VA: Teachers of English to Speakers of Other Languages.
Savignon, S. (1983). Communicative competence: Theory and classroom practice. Reading, MA: Addison-Wesley.
Spohn, J., & Zafft, C. (2006). ABE-to-college transitions and the ELA framework: Extending the curriculum. Field Notes. 15(4), 19-20.
The Splendid ESOL Web, Pima College Adult Education http://cc.pima.edu/~slundquist/index.htm
Stites, R. (1999, September). A user’s guide to standards-based educational reform: From theory to practice. Focus on Basics, 3(C), 1, 3-7.
Teachers of English to Speakers of Other Languages. (2001). Scenarios for ESL standards-based assessment. Alexandria, VA: Author.
Tomlinson, B. (2003). Developing materials for language teaching. Middlesex, UK: Continuum International Publishing Group.
Weddel, K. S., & Van Duzer, C. (1997). Needs assessment for adult ESL learners. Washington, DC: Center for Applied Linguistics. Available at www.cal.org/caela/esl_resources/digests/Needas.html
Weiss, I., Pasley, J., Smith, P., Banilower, E., & Heck, D. (2003). A study of K-12 mathematics and science education in the United States. Chapel Hill, NC: Horizon Research, IN. Retrieved December 27, 2006, from www.horizon-research.com/reports/2003/insidetheclassroom/highlights.php
Wiggins, G., & McTighe, J. (1998). Understanding by design. Alexandria, VA: Association for Supervision and Curriculum Development.
Young, S., & Smith, C. (2006). Understanding adult ESL content standards. Washington, DC: Center for Adult English Language Acquisition. Available at http://www.cal.org/caela/esl_resources/briefs/contentstandards.html
Adult Education Content Standards Warehouse www.adultedcontentstandards.ed.gov/
This Web site gives assistance to adult educators in their efforts to develop, align, and implement content standards in English language acquisition, mathematics, and reading.
Blaz, D. (2000). A collection of performance assessment tasks for foreign language. Larchmont, NY: Eye on Education.
A collection of creative strategies for assessing learner performance in the four main areas of foreign languages: speaking, listening, reading, and writing
National Institute for Literacy's Adult Education Content Standards Discussion List www.nifl.gov/mailman/listinfo/Contentstandards
Archives from a former discussion list about developments in and resources for use of adult education content standards
The Nevada Literacy Web site
This Web site describes Nevada’s ESL/EL Civics standards and shows how Nevada aligns its standards to objectives, CASAS competencies, and program resources.
A Web site for using and building graphic organizers, rubrics, and supports specifically for writing
The Standards section of the Adult Literacy Wiki http://wiki.literacytent.org/index.php/Standards This Web site includes information about online resources, Web sites, and standards implementation in the United States and other English-speaking countries.
This document was produced by the Center for Adult English Language Acquisition (CAELA) at the Center for Applied Linguistics ( 4646 40th Street, NW, Washington, DC 20016, 202-362-0700) with funding from the U.S. Department of Education (ED), Office of Vocational and Adult Education (OVAE), under Contract No. ED-04-CO-0031/0001. The opinions expressed do not necessarily reflect the positions or policies of ED. This document is in the public domain and may be reproduced without permission.