About CIERA
The Center for the Improvement of Early Reading Achievement (CIERA) is the national center for research on early reading and represents a consortium of educators in five universities (University of Michigan, University of Virginia, and Michigan State University with University of Southern California and University of Minnesota), teacher educators, teachers, publishers of texts, tests, and technology, professional organizations, and schools and school districts across the United States. CIERA is supported under the Educational Research and Development Centers Program, PR/Award Number R305R70004, as administered by the Office of Educational Research and Improvement, U.S. Department of Education.
CIERA's mission is to improve the reading achievement of America's children by generating and disseminating theoretical, empirical, and practical solutions to persistent problems in the learning and teaching of beginning reading.
The model that underlies CIERA's efforts acknowledges many influences on children's reading acquisition. The multiple influences on children's early reading acquisition can be represented in three successive layers, each yielding an area of inquiry of the CIERA scope of work. These three areas of inquiry each present a set of persistent problems in the learning and teaching of beginning reading:
CIERA Inquiry 1
Readers and Texts
Characteristics of readers and texts and their relationship to early reading achievement. What are the characteristics of readers and texts that have the greatest influence on early success in reading? How can children's existing knowledge and classroom environments enhance the factors that make for success?
CIERA Inquiry 2
Home and School
Home and school effects on early reading achievment. How do the contexts of homes, communities, classrooms, and schools support high levels of reading achievement among primary-level children? How can these contexts be enhanced to ensure high levels of reading achievement for all children?
Policy and professional effects on early reading achievement. How can new teachers be initiated into the profession and experienced teachers be provided with the knowledge and dispositions to teach young children to read well? How do policies at all levels support or detract from providing all children with access to high levels of reading instruction?
An Analysis of Early Literacy Assessments Used for Instruction
CIERA Report #2-013
Samuel J. Meisels and Ruth A. Piker
University of Michigan
CIERA Inquiry 2: Home and School
What classroom-based literacy measures are available to teachers and how can we best characterize the instructional assessments teachers use in their classrooms to evaluate their students' literacy performance?
CIERA April 23, 2001
This report focuses on results of a systematic study of instructional assessments of early literacy designed by teachers and other educators for use in K-3 classrooms. The report presents the methodology and coding scheme used for collecting classroom-based measures and evaluating their content. It provides data about how reading and writing skills are assessed by teachers and shows the relationship between the skills included on these assessments and the skills associated with national standards and benchmarks. It also characterizes the instructional assessments teachers use in their classrooms to evaluate their students' literacy performance in terms of categories of skills assessed, types of assessment models utilized, differences in student responses elicited by the assessments, forms of administration, types of mental processing required of students, and other parameters. The discussion concerns questions about the psychometric properties of these assessments, their relationship to national standards, and their place in the instructional process for classroom teachers.
University of Michigan School of Education
CIERA
610 E University Ave., Rm 1600 SEB
Ann Arbor, MI 48109-1259
734.647.6940 voice
734.615.4858 fax
ciera@umich.edu
©2001 Center for the Improvement of Early Reading Achievement.
This research was supported under the Educational Research and Development Centers Program, PR/Award Number R305R70004, as administered by the Office of Educational Research and Improvement, U.S. Department of Education. However, the comments do not necessarily represent the positions or policies of the National Institute of Student Achievement, Curriculum, and Assessment or the National Institute on Early Childhood Development, or the U.S. Department of Education, and you should not assume endorsement by the Federal Government.
An Analysis of Early Literacy Assessments Used for Instruction
Samuel J. Meisels and Ruth A. Piker
University of Michigan
T he current administration in Washington has made development of early reading skills a topic of great importance. President Bush's predecessor, Bill Clinton, did his part to raise early reading assessment to a pinnacle of public attention when, in his 1997 State of the Union address, he said that "Every state should adopt high national standards, and by 1999 every state should test every fourth grader in reading and every eighth grader in math to make sure these standards are met. . . . Good tests will show us who needs help, what changes in teaching to make, and which schools to improve."
Unfortunately or not, the President's words outstripped reality. Congress fought his plan for "voluntary" national tests in reading and math and refused to allow government funds to be used for this purpose. On a more academic level, one can see that his goals for "good tests" can never be achieved by a single assessment: No test can, by itself, serve as many purposes as the President desired. First, in order for a test to "show us who needs help" we would need information about individuals that predicts future performance. This is what Resnick and Resnick (1992) call selection and certification of students. Second, in order to know what changes in teaching to make, we would need to have tools available that would permit us to diagnose particular strengths and weaknesses in individual student performances and then be in a position to monitor the effects of instruction. This type of assessment is called instructional management and monitoring, or instructional assessment. Finally, if we want our tests to tell us "which schools to improve" we are seeking an assessment that provides public accountability and program evaluation. Such tests provide those with responsibility for the funding and supervision of education with information on whether a particular program is succeeding in its academic goals (Resnick & Resnick, 1992).
In short, no single assessment can cover all of the purposes that are required of tests and evaluations. Of all the testing that take place in schools, the vast majority is created by teachers or is otherwise some form of informal classroom or instructional assessment (Stiggins & Bridgeford, 1985; Stiggins, Griswold, & Wikelund, 1989). Although teachers devote some attention to diagnostic assessments in order to enhance their instructional practices (see Lipson & Wixson, 1991; Murphy, Shannon, Johnston, & Hansen, 1988), and schools, districts, states, and the federal government certainly impose accountability testing in great quantities (see Anthony, Johnson, Mickelson, & Preece, 1991; Calkins, Montgomery, Santman, & Falk, 1998), the vast majority of the available assessment time and energy is consumed by instructional assessment.
We define instructional assessment as formal or informal methods of obtaining information about children's classroom performance in order to guide instructional decision-making and provide instructionally relevant information to teachers. In an instructional assessment the primary focus is on individual learning rather than on group reporting of average scores. More specifically, instructional assessment is not designed to rank or compare students or to be used for high-stakes purposes. Rather, it is a tool for the teacher, and its value is linked directly to its impact on instruction. Instructional assessments are intended to clarify what students are learning and have begun to master by providing information that is relevant to understanding individual students' learning profiles. In this way, like other authentic performance assessments, their purpose is to enhance learning and improve instruction (Calfee, 1992; Calfee & Hiebert, 1991; Meisels, 1997).
Conventional standardized tests of reading achievement have been subjected to extensive analysis (see Haladyna, Nolen, & Haas, 1991; Stallman & Pearson, 1990a, 1990b), but less information is available regarding instructional assessments. Indeed, the National Research Council's Committee on the Prevention of Reading Difficulties (Snow, Burns, & Griffin, 1998) made the following recommendation:
Toward the goal of assisting teachers in day-to-day monitoring of student progress along the array of dimensions on which reading growth depends, the appropriate government agencies and private foundations should sponsor evaluation, synthesis, and, as necessary, further development of informal and curriculum-based assessment tools and strategies. In complement, state and local school districts should undertake concerted efforts to assist teachers and reading specialists in understanding how best to administer, interpret, and instructionally respond to such assessments. (p. 337)
In short, notwithstanding several attempts to describe the significance and role of instructional assessment in the classroom routine (Taylor, 1990; Valencia & Calfee, 1991; Winograd, Paris, & Bridge, 1991), more focus is needed on the area of instructional assessment--particularly in the area of literacy. This technical report is intended to provide a compilation and analysis of early literacy assessments used for instruction.
The purpose of this study is threefold: (a) to gain an understanding of classroom-based literacy measures that are available to teachers; (b) to characterize the instructional assessments teachers use in their classrooms to evaluate their students' literacy performance; and (c) to learn more about how teachers assess reading and writing elements. Throughout this report we will refer to "skills and elements" to denote what the literacy assessments are designed to measure. In some cases (e.g., spelling, punctuation, phonetic analysis), the assessments focus clearly on skills. In other cases (e.g., demonstrating concepts of print; extracting meaning from text; assessing self-reflection, motivation, or attitudes), the term "literacy element" is more appropriate.
Our specific research questions focus on both the measures available for analysis and the skills and elements inherent in the measures. Regarding the measures, we asked the following questions:
- With which grades and languages are the measures used?
- How often are teachers supposed to administer these assessments, and in what format (individual or group)?
- Which elements are included most frequently in the measures?
- What are the psychometric properties of the assessments we analyzed?
- What is the relationship of national compilations of standards (specifically those from the Mid-continent Regional Educational Laboratories [McREL]) to the measures?
- What type of measurement methods are used most frequently, and what types of student responses and mental processing are the assessments designed to elicit?
Regarding the literacy skills or elements that are implicit in the measures:
- Which skills or elements occur most frequently in the assessments?
- What is the match between national compilations of standards (e.g., McREL) and the elements we analyzed?
- What is the distribution of skills and elements by grade and form of administration?
- What is the most frequent method of assessment of particular elements?
- What is the most common format used for assessing skills or elements and the most frequent type of student response and mental processing for assessing each element?
This report presents our response to these research questions as well as a set of recommendations based on them. It is accompanied by a database available on the CIERA website (www.ciera.org) that provides detailed information about each of the assessments reviewed for this report.
A. Sample
1. Selection criteria
We used four criteria to select early literacy assessments for this study. First, we included measures that were developed for use in classrooms by teachers, school districts, state departments of education, and/or researchers. As will be described later, these measures were nominated by teachers and other educational professionals. Second, for the most part we focused on measures that were developed and distributed by noncommercial publishers. Third, we included measures whose primary purpose was instruction, rather than accountability. Finally, we examined assessments that targeted children between kindergarten and third grade. Measures that extended beyond third grade were only analyzed to grade 3.
Several measures that were recommended by our sources were not included in our sample. We excluded measures designed primarily for toddlers, preschoolers, or students in fourth grade and beyond; non-literacy related assessments (e.g., science, social studies); assessments used for research purposes; and assessments primarily used for accountability purposes. We included, but did not comprehensively sample, measures that assess motivation, self-perception, and attitudes toward reading.
2. Sources of measures
We gathered the measures used in this survey from five sources: listservs, personal contacts, literature searches and published reviews of the measures, websites, and newsletter postings. We posted a request for information regarding classroom-based literacy practices on eight listservs (see Table 1). These listservs reach a wide range of practitioners, researchers, and policymakers, many of whom provided us with names of informal literacy assessments and with referrals regarding people to contact, books to review, and websites to examine.
Personal contacts took place with practitioners, researchers, state-level policymakers, and representatives of professional reading organizations. These contacts included individuals who responded to our listserv postings as well as leading researchers, state reading coordinators, academics, and others who were recommended to us. These conversations led to our receiving copies of several measures, as well as additional suggestions for other literacy assessments.
|
American Educational Research Association--Measurement and Research Methodology |
||
Our literature search identified numerous books, journals, articles, and papers that were reviewed for relevant assessment information. Most sources consisted of guidelines for developing informal assessments, assessing students in higher grades, and current trends in the field of assessment. A few included specific assessments for K-3. The majority of the assessments were found in books, and several were located in such reading journals as The Reading Teacher and Elementary School Journal. Other searches provided standardization and psychometric properties for the assessments we received.
We also accessed the websites of numerous national organizations, state departments of education, schools, and the U. S. Department of Education's Cross-Site Index (see Table 2). These websites were primarily concerned with assessment-related information and described articles, books, and handouts with guidelines for developing informal assessments. The few sites with specific literacy assessments for K-3 described materials that were commercially developed and distributed.
We posted a notice in a large number of local, state, and national newsletters that reach reading teachers and early childhood and elementary educators. Local affiliates of the Michigan Reading Association, state affiliates of the National Association for the Education of Young Children (NAEYC), and affiliates of the International Reading Association agreed to post our notice in their newsletters (see Table 3). Although these requests for literacy assessments reached a large number of practitioners, we received only a handful of assessments from this effort. However, the measures we received included references to other literacy-related measures for K-3. Nevertheless, it is clear that this report does not include an exhaustive enumeration of informal literacy assessments. It represents strictly a sampling of the universe.
3. Developers and currency of measures
Overall, we collected a large number of measures (N = 89) that were created by a wide spectrum of developers (states, 10%; districts or schools, 11%; teachers, 16%; researchers, 60%; and other developers, 3%). The copyright dates of the assessments extend from 1936-1999, although the majority are from the past 10 years
(N = 60). For assessments with more than one version, the most recent edition was analyzed. All measures were examined directly, either through obtaining copies of the measures from the developers or through library or interlibrary loan requests.
The coding scheme for analyzing the measures is adapted from Stallman and Pearson (1990b), Pearson, Sensale, Vyas, and Kim (1998), Stiggins (1995), Mariotti and Homan (1997), and our own explanatory analysis. The list of analytic categories is presented in Table 4. The coding scheme is organized around the types of literacy elements evaluated and the ways in which these skills or elements are assessed at different grade levels. The scheme is divided into two broad sections: (a) general overview, and (b) skills or elements tested, with each section further subdivided into more discrete elements. The coding manual, which provides a description of each section, is located in Appendix A. Below we describe the contents of the coding scheme.
|
I. Amount of time required to administer K. Format(s) for recording student response |
1. General overview
The general overview contains identifying information about the measure, including names of authors, general availability, overall purpose, and language availability. The purpose of the measures indicates its overall intent. Some measures are very specific about the types of elements they evaluate (e.g., spelling, phonemic awareness), whereas others are more global and encompass a range of elements (e.g., reading, writing). Information concerning the measure's standardization and psychometric properties is located in this section. Finally, any additional information unique to the measure that is not included in the Skills or Elements Tested section is indicated in the comments section. The general overview also provides a summary of the contents of the Skills or Elements Tested section, the grade levels evaluated by the measure, the form of administration, frequency, time required to administer the measure, assessment models, format for recording student responses, and category of elements.
This section examines the specific skills or elements the measures are designed to assess. Eighty-eight percent of the measures assess more than one literacy element, ranging from 1 to 67 different elements.
The elements are divided into eleven literacy-related categories, with two additional categories examining students' oral language and other elements. These categories are further subdivided into specific constituents, accounting for 133 skills or elements in all (see Table 5). The categories and constituent elements were derived from our analysis of the assessments. We compared these elements to the standards and benchmarks compiled by the Mid-continental Regional Educational Laboratory (McREL; Kendall & Marzano, 1997). McREL standards and benchmarks provide a format that reflects state and national standards in the various curriculum domains. The McREL content standards for Language Arts comprise eight standards for K-12. We include the eight Language Arts standards with their benchmarks for K-3 as an Appendix to the coding manual (see Appendix A) and we indicate with an asterisk those elements that are referenced in the McREL content standards.
|
a. Illustrations Are Representative of the Story c. Types of Compositions 1 d. Uses Illustrations to Express Ideas e. Uses Lively and Descriptive Language f. Use of Formal and/or Literary Language h. Writing Attends To Audience See Matches a McREL Benchmark and Standard. j. Writing Contains A Purpose See Matches a McREL Benchmark and Standard. k. Writing Contains Description and Details l. Writing Conveys a Sense of Story m. Writing Has Evidence of Beginning, Middle, and End n. Writing Is Easy to Understand And Follow o. Writing Is Logical And Sequential p. Writing Process See Matches a McREL Benchmark and Standard. |
|
|
a. Capitalization See Matches a McREL Benchmark and Standard. b. Directional Principles in Writing c. Grammatically Correct Sentences See Matches a McREL Benchmark and Standard. c. Linguistic Organization See Matches a McREL Benchmark and Standard. d. Paragraphs See Matches a McREL Benchmark and Standard. e. Punctuation Marks See Matches a McREL Benchmark and Standard. f. Spelling See Matches a McREL Benchmark and Standard. g. Uses Complex Word Structures |
|
|
b. Directionality See Matches a McREL Benchmark and Standard. c. Identification of Parts of a Book See Matches a McREL Benchmark and Standard. e. Letter and Word Order See Matches a McREL Benchmark and Standard. g. Understands Punctuation Marks h. Understands That Print Conveys Meaning See Matches a McREL Benchmark and Standard. i. Understands Upper- And Lower-Case Letters j. Word Boundaries See Matches a McREL Benchmark and Standard. |
|
|
a. Decoding Words See Matches a McREL Benchmark and Standard. b. Identification of Beginning Sounds* e. Phonemic Awareness See Matches a McREL Benchmark and Standard. |
|
|
f. Reading Accuracy See Matches a McREL Benchmark and Standard. h. Reads as if Passage is Meaningful i. Texts Student Can Read See Matches a McREL Benchmark and Standard. |
|
|
b. Monitoring Own Reading Strategies See Matches a McREL Benchmark and Standard. c. Self-Correction See Matches a McREL Benchmark and Standard. d. Using Pictures and Story Line for Predicting Context and Words See Matches a McREL Benchmark and Standard. |
|
|
a. Comments on Literary Aspects of the Text b. Connects Universally Shared Experiences With Text See Matches a McREL Benchmark and Standard. c. Distinguishes Fantasy From Realistic Texts See Matches a McREL Benchmark and Standard. e. Identify Cause-Effect Relationships f. Inferences See Matches a McREL Benchmark and Standard. h. Literary Analysis See Matches a McREL Benchmark and Standard. i. Prediction Strategies See Matches a McREL Benchmark and Standard. j. Provides Supporting Details See Matches a McREL Benchmark and Standard. k. Reference to Evidence Presented in Text l. Retelling See Matches a McREL Benchmark and Standard. |
|
|
c. Family Support and Prior Experience |
|
|
a. Familiarity With Types of Texts b. Monitoring How Student Reads g. Self Assessment in Non-Language Arts Domains |
|
|
a. Attitudes Towards Other Literacy Activities |
|
|
i. Participates in Group Discussion k. Responses Make Connections to the Situation |
|
We gathered information about the grade of the student for which the element is intended; different elements may be evaluated in different grades by the same measure. Certain elements are more relevant to earlier grades, such as letter identification and identification of parts of a book, whereas other elements may be more specific to older children in second or third grade, such as writing in paragraphs and using complex sentence structures. The form of administration--whether the assessment uses an individual, one-to-one setting, a group format, or both--is noted next. Several forms may be used for different elements within the same measure. The frequency and amount of time required to administer this part of the measure is also noted for each element. This helps us understand how often teachers evaluate elements, and specifically which elements are evaluated regularly and which are assessed infrequently. The amount of time teachers spend evaluating students' literacy elements in a one-to-one setting or in a group suggests how much time is spent on the assessment process.
The six assessment models in the coding scheme are based in part on the work of Stiggins (1995): (a) clinical interviews, (b) constructed response, (c) observation, (d) on-demand response (also described as closed-response set), (e) student self-assessment, and (f) multiple responses (see Table 6). The first four and the sixth of these models emerged from our readings and a priori categorizations; however, student self-assessment was derived from the data we reviewed. Teachers, researchers, and districts view students' involvement with the evaluation of their work as a growing and critical aspect of the assessment process. We also found through our analyses that the same element was sometimes evaluated differently with the same tool. In cases in which a element is assessed in multiple ways, we classified the model as comprising multiple responses.
Item response format covers a list of formats that practitioners use for recording student responses (see Table 7). The formats were derived from several sources, including Stallman and Pearson (1990b) and Pearson et al. (1998), as well as from our analysis of the measures we obtained. Stallman and Pearson (1990b) only included checklists and multiple choice. Pearson et al. (1998) expanded Stallman and Pearson's (1990b) analysis to include four more categories. We further expanded the categories to include twelve formats and we renamed the formats to distinguish among the numerous types of formats available to practitioners.
The number of items the measure offers for evaluating a specific skill or element describes the quantity of information teachers are asked to gather in order to assess a particular element. However, the number of items says very little in itself; a place is provided for a description of the items, such as "uses a passage or rubric," "is a question or statement," or "is part of a larger checklist or questionnaire."
The presentation section uses subcategories from Stallman and Pearson (1990b), with revisions from Pearson et al. (1998). The mode of presentation, which contains six options, describes the main mode of presentation used by the examiner, including auditory and visual (see Table 8). The unit of presentation is the type of stimulus to which the student is asked to respond; we added a few options and eliminated others to arrive at a total of 24 options (see Table 8). Examples of units of presentation that emerged from our data include books, connected discourse, letters, phonemes, stories, and words.
The response section is also borrowed from Stallman and Pearson (1990b), with revisions by the authors and by Pearson et al. (1998). Specific types of student responses are divided into three subcategories: type of mental processing, unit of response, and student response (see Table 9). The type of mental processing describes how students process the information presented in order to provide the appropriate response. We added three additional options to the original options of identification, production, recognition, and other: recall, reproduction, and multiple responses. Recall is common when assessing comprehension; however, reproduction rarely emerged. The unit of response refers to the stimuli used by the student to indicate the correct answer to the item. Examples of stimuli used by the measures we collected include grapheme, objects, phrase, picture, punctuation marks, and sounds. The student's response categorizes what the student does when responding to the item.
Finally, we indicated how the element is scored (rating scale, rubric, or yes/no). In the notes section we include any additional information relevant to the element.
We present frequencies to describe the general overview of the measures we collected, including grade levels, forms of administration, types of assessment models, formats for recording student responses, and categories of elements. The frequencies offer a clear description of the measures. The next step of the analysis focuses on the elements evaluated by the measures, including the methodology, formats, grade levels, and student responses to the items. We also perform cross-tabulations of elements by assessment models, student response formats, and response types. In addition, we examine the standardization and psychometric data that are available concerning these measures. Finally, we provide a description of two samples of our measures in order to demonstrate the kind of information available in the database. The two measures are Guidance in Story Retelling (Morrow, 1986), and Literacy Assessment for Elementary Grades (St. Vrain Valley School District, 1997). The format used to describe these measures was applied to all of the assessments we collected.
This section is divided into two parts. First, we present analyses by specific assessments. In the second part we focus on elements and provide analyses that cut across our entire sample of assessments.
Our analysis includes 89 assessments. A brief overview of the 89 measures is presented in Appendix B; a comprehensive review of each measure is available at www.ciera.org. The summary provides the name of the assessment, author, purpose, grade, form of administration (individual or group setting), and the category of elements each measure assesses. The name of the measure is the title of the tool or the title of the group of measures developed by the same author(s). The groups of measures are placed under the umbrella of the author or title of the book. For example, An Observational Survey (Clay, 1998) contains several tools, such as Concepts about Print and Dictation; all of these assessments are found under the title of Clay's book. Many measures state their purpose as part of the measure. Some descriptions are global, such as "evaluates students' literacy development" (MacArthur CCDP Follow-up Study, 1998), whereas others are very specific, for example "to estimate students' reading level, group students effectively, and appropriately choose textbooks, and to plan intervention instruction" (Leslie & Caldwell, 1995). Measures that do not have a stated purpose receive a generic statement of "to evaluate students' reading and writing abilities."
The grades the measures are to be used with range from K-3; the distribution is presented in Figure 1. Only 10% (N = 9) of the measures are designed for a particular grade level. Many apply to students in two or three grades (N of two grades = 16; N of three grades = 24), with almost half of the measures evaluating literacy elements at all four grade levels (N = 40).
All measures are available in English, and only 5% (N = 4) are available in Spanish (one assessment is available in Danish; see Table 10). Seventy percent of the measures are designed for individual administration, rather than for use in a group setting. These individual forms of administration also include teacher observations of students. Measures that ask teachers to use observations of students in order to complete a checklist are coded as individual administrations unless the measure states that the teacher can complete the checklist or rubric within a group setting. Only 7% of the measures we collected are intended to be administered solely to a group of children.
Table 10 shows how often the measures indicate exactly when to administer the entire assessment or parts of the measure. Fewer than half of the measures (44%) we analyzed explicitly state the minimum number of times that a teacher should evaluate students' literacy elements. About a quarter of the measures (26%) indicate the length of time required to complete the evaluation.
Language |
||
Administration |
||
The skills or elements evaluated by the assessments range across 13 categories (see Table 11). Of the assessments we collected, all categories are represented in at least 26% of the measures. More than half of the assessments evaluate students' use of conventions, phonics, reading, and comprehension elements. Evaluations of writing process, print awareness, and reading strategies appear somewhat less frequently (42-48%). The other six categories are included in one third of the assessments. A summary of the specific elements assessed by each measure is presented in Appendix C.
Next, we examine the number of McREL standards found throughout our measures. Table 12 indicates the number of assessments with one or more standards, up to all eight standards. One or two McREL standards are represented in nearly one third of the measures (N = 28), and 13% (N = 12) of the assessments contain a element relevant to all 8 standards. Only 4% (N = 4) of the measures do not contain any McREL standards. The specific standard that is represented most frequently is Standard 5 ("Demonstrates competence in the general skills and strategies of the reading process"). Seventy-three of the assessments included this standard.
We also investigated the various methodologies represented by the assessments. Of the 89 measures, more than half (N = 47) use two very different approaches--observation or on-demand methods--for evaluating students' literacy skills (see Figure 2). Only 29% (N = 26) use constructed responses, and such responses occur predominantly with the writing process and conventions; 16% (N = 14) provide students with the opportunity to participate in the evaluation of their work. Observation, constructed response, and on-demand methods are used most consistently across all grade levels.
All twelve item formats are used across the measures to record student responses (see Table 13). Of the 89 measures, 42% (N = 37) use oral-directed responses as part of their assessment. The next most common format is checklist (36%, N = 32), followed by written open-ended (18%, N = 16). The item formats used by the measures are related to the methodologies; only checklists are used by all methods. An observation methodology in conjunction with checklists is the most frequent combination.
We further explore student responses to the assessments by examining mental processing strategies. The most common type of mental processing used
by students for processing the information presented is identification (N = 50, 56%; see Table 14). Production, recall, and "other" are the next most common types of mental processing required of students by the assessments, followed by recognition and multiple responses. The table demonstrates that students use 10 different ways to respond to the items. More than 60% of the measures require students to respond orally; this is followed by written responses (46%). Of the 10 possible ways of responding included in our analysis, 5 were rarely used, occurring in less than 10% of the assessments.
B. Analysis by Skills or Elements
This section describes our analyses in terms of the constituent skills or elements of the assessments. Each skill or element (N = 133) appears only once for each assessment in our coding scheme, regardless of the multiple ways it may be assessed. The frequency of a single element appearing across all assessments ranged from 1-41; a summary of the elements that appear on 10 or more measures is presented in Table 15. The specific skill of decoding words appeared in more than 40 measures; the next most common skill was spelling (N = 38), followed by reading accuracy, summarizing main ideas, and providing supportive details (N for each = 32). In short, this table shows us which elements appear most frequently in the 89 measures we analyzed. (For an analysis of the number of elements included in each assessment, see Appendix C.)
|
Using Pictures and Story Line for Predicting Context and Words |
||
We examined the number of constituent skills or elements that match a particular standard on the McREL standards in the Language Arts content area. We found that 27% (N = 55) of our elements were represented in the McREL standards; Figure 3 shows the number of elements associated with each standard. Overall, we identified a total of 133 constituent elements that were included in the 89 assessments. In addition to the 55 that match the McREL standards, 25 (19%) reflect motivation, self-perception, metacognition, and attitude towards reading. The remaining elements (N = 52; 39%) do not match the McREL standards or the motivation/self-perception group. The three groups of elements are presented in Appendix D.
We further analyzed the distribution of grade levels and forms of administration by constituent skills or elements. Ninety-two percent (N = 123) of the elements are assessed in all grades, K-3. The elements that are not evaluated in all grades are part of the motivation, self-perception, attitude, and metacognition categories (N = 10). These elements tend, on average, to be evaluated in second and third grades, when they are more stable. The form of administration (individual or group) for evaluating the skills or elements is presented in Figure 4. Almost all of the skills or elements are assessed individually, with two thirds assessed as either individual or group.
The most common methodology used for evaluating a particular skill or element is observation (N = 123; see Figure 5). Half of the elements were assessed using either constructed response (N = 67) or on-demand response (N = 65). The least frequently used methodology was clinical interview (N = 20), which is most commonly associated with motivation, self-perception, attitude, and metacognition elements.
The item formats used by administrators for recording student responses across skills or elements are presented in Table 16. Elements are recorded most often with checklists (N = 117). The next most frequently used method of tracking student responses is observation (N = 92), followed by multiple responses, written open-ended, oral-directed, written-directed, and informal reading inventory.
In Table 17 we examine the specific type of response students use to identify correct answers and what the student does in response to each item with the constituent skills or elements. For 90% of the elements (N = 120), teachers decide which activity to use in order to assess a particular skill or element. Approximately two thirds of the elements (N = 89) call upon students to respond in multiple forms and to produce the correct response in order to show their mastery of a skill or element. The use of identification is limited to half of the elements (N = 68). Students respond in 10 different ways when indicating the correct answer; Table 17 lists those responses that occur with more than 10% of the skills or elements. The responses with fewer than 10% include draw, find, manipulate, and mark.