Beating the Odds in Teaching All Children to Read
Barbara M. Taylor, University of Minnesota
P. David Pearson, Michigan State University
Kathleen F. Clark, University of Minnesota
Sharon Walpole, University of Virginia
W e all want the best schools possible for our children, schools that help them acquire the knowledge, skills, and dispositions they will need to pursue whatever dreams and paths they wish. Yet the reality is that many of our children are not reading well enough to keep up with the demands of school (Campbell, Donahue, Reese, & Phillips, 1996; Donahue, Voelkl, Campbell, & Mazzeo, 1999), let alone the demands of our society or their personal dreams. In the recent national report, Preventing Reading Difficulties in Young Children, a National Academy of Science Committee concluded that "quality classroom instruction in kindergarten and the primary grades is the single best weapon against reading failure" (Snow, Burns, & Griffin, 1998). The committee recommended that our number one priority for funding research should be to improve classroom reading instruction in kindergarten and the primary grades.
In a recent report of a three-year study of schools implementing special strategies to improve reading achievement, researchers described classroom instruction as "in one sense, distressing" (Stringfield, Millsap, & Herman, 1997, p. 2). In the elementary schools, instruction was predominantly teacher-led, focused on discrete skill instruction, and driven by management concerns. There were relatively few observations of students engaged in sustained reading or students applying what they were learning. On the other hand, Stringfield, Millsap, and Herman pointed out that even in schools nominated as exemplary, there was ample room for instructional improvement, which would, if implemented, lead to greater gains in reading achievement.
In addition to advocating improved classroom reading instruction, the Committee on the Prevention of Reading Difficulties in Young Children discussed the importance of systematic, schoolwide restructuring efforts in reading. The committee (Snow et al., 1998) recommended that poor performing schools consider reading reform efforts with a dual focus on schoolwide organizational issues and improved classroom reading instruction.
In their special strategies study, Stringfield et al. (1997) found that reform programs that focused on the primary grades had larger achievement gains in reading than schools that spread their efforts out across the elementary grades or into the secondary grades. They also found that schools that adopted externally developed programs had greater achievement gains than schools that developed their own programs.
In yet another recent report of a large national study of 400 Chapter 1 schools, researchers found that higher levels of poverty, greater application of grade retention policies, and higher levels of student disciplinary actions were related to lower student achievement (Puma et al., 1997). Only 5 schools in the pool of 400 were identified as exceptional. These schools tended to have a "more experienced principal, a schoolwide Chapter 1 program, some tracking by ability in grades 1-6, lower rates of teacher and student mobility, a balanced emphasis on remedial and higher-order thinking in classroom instruction, and higher levels of community and parent support" (p. 62). Except for first grade, in which grouping was used, whole-class instruction was the dominant practice in all schools.
These three recent national reports highlight the importance of and need for additional research on schools that serve the needs of poor children by increasing their achievement and, hence, their educational opportunities. The purpose of the present study was to examine the instructional and organizational factors that might explain how and why some schools across the country are beating the odds by attaining greater than expected primary-grade reading achievement with populations of students at risk for failure by virtue of poverty. We pause to emphasize the terms instructional and organizational, for it is our belief that only when we attend to both school-level (organizational) and classroom-level (instructional) facets of reform do we meet our aspirations.
Within this broader framework, we were, like the researchers in the special strategies study (Stringfield et al., 1997), interested in both imported models of reform (where they had adopted an external intervention program or school reform program) and homegrown reform efforts. To that end, we sought schools in both categories. Among the reforms imported were schoolwide programs and both tutorial and small-group interventions for at-risk youngsters. As it turned out, we also had combination reform efforts; in several schools, for example, an imported intervention program was set within a homegrown schoolwide reform effort.
Effective Schools
Research on effective schools relevant to reading achievement, much of which was conducted in the 1970s and early 1980s, was documented in a review entitled "Teacher and School Effects in Learning to Read" by Hoffman (1991) in the Handbook of Reading Research, Volume II. Hoffman described eight attributes of effective schools frequently summarized in the literature (e.g.. Shavelson & Berliner, 1988), including:
- A clear school mission;
- effective instructional leadership and practices;
- high expectations;
- a safe, orderly, and positive environment;
- ongoing curriculum improvement;
- maximum use of instructional time;
- frequent monitoring of student progress; and
- positive home-school relationships.
In a study of 4 outlier inner-city schools, Weber (1971) found strong leadership, high expectations, positive school climate, strong emphasis on reading, and continuous evaluation of pupil progress related to the identified school reading success criteria of median achievement level on a normed standardized reading achievement test and having a relatively small number of children with serious reading difficulties. In a study of 5 schools found to be most effective out of a sample of 741 schools which were part of a study of compensatory reading programs, Wilder (1977) found the following factors common to all 5 schools: reading was identified as an important instructional goal; leadership in the reading program was provided by either the principal or reading specialist; attention was given to basic skills; a breadth of materials was made available; and ideas were communicated across teachers, a process which was typically fostered by the program leader.
In a more recent, longitudinal study on schools implementing special strategies for educating disadvantaged children, Stringfield et al. (1997) found that the schools demonstrating the greatest achievement gains worked hard at both initial implementation and long-term maintenance of an innovation. But the researchers also noted the importance of systematic self-improvement in these schools, in which the innovations continued to evolve and expand. Externally developed research-based programs and programs that focused on whole school reform were related to greater achievement gains than locally developed programs and innovations composed of various pull-out programs. The study also found support for the premise that students placed at risk of academic failure could achieve at levels that met national averages.
In a study of five effective Title I schools, Puma et al. (1997) found that high-performing, high-poverty schools had lower than average teacher and student mobility, principals with more years of experience, and more orderly school environments than average high-poverty schools. Better school climates and better relations with administration and the community were also reported, as well as greater parent involvement and more parents with high expectations for their children's future educational attainment. All of these high-performing schools had tracking by ability in grades 1-6. In three of the five schools, teachers emphasized remedial and higher order comprehension skills in reading.
Although research on effective schools has been favorably received by school leaders and policymakers, Hoffman (1991) points to limitations of this research stemming from its lack of connection to classroom practice and to insufficient information on the process schools went through to become effective. Even so, that the same characteristics arise time and time again has led many reformers to suggest that these findings ought to be translated into policies that can guide reform.
Effective Teachers
In addition to research on effective schools, Hoffman (1991) summarized a considerable body of research, spanning the 1960s through the 1980s, on teachers who were exceptionally effective in helping students learn to read. Hoffman reports on a literature review of effective teaching by Rosenshine and Furst (1973) in which they found several teacher behaviors consistently related to student achievement: clarity, variability, enthusiasm, task orientation, teacher directness, student opportunity to learn criterion material, use of structuring comments, multiple levels of questions, and criticism (which was negatively related to achievement).
In a study of the achievement of students of 165 second- and third-grade teachers conducted over a three-year period, Brophy (1973) reported on the patterns of the most effective teachers, who represented about a third of the sample. He found that the most effective teachers were businesslike with a strong sense of task and direction for themselves and their students, had high expectations for their students' achievement, and redoubled efforts when failure was experienced, especially in low socioeconomic status (SES) environments. The most effective teachers had strong management skills, but their classrooms were not stern or oppressive. They had high levels of pupil engagement and were proactive in preventing disruptions. The most effective teachers engaged in the practice of probing individuals when incorrect responses were offered instead of simply calling on someone else or giving the answer themselves. The students in low-SES classes of the most effective teachers had a success rate of about 80% correct when answering teacher-directed questions, almost all of which were literal. In a follow-up intervention study of first-grade teachers engaged in small-group instruction, Anderson, Evertson, and Brophy (1979) found that greater achievement was related to more time spent in reading groups, more active instruction, shorter transitions, introduction of lessons with an overview, and follow-up by teachers to incorrect responses with attempts to improve upon them.
In a study of 166 first- and third-grade teachers of children who had been in Head Start, variables positively related to gains in reading included time spent in academic activities, frequency of small-group instruction in basic skills, and frequency of supervised seatwork activities (Stallings & Kaskowitz, 1974). The lowest SES students benefited most from intense, small-group instruction.
In a study of 25 second-grade and 21 fifth-grade classrooms, Fisher et al. (1980) found that the more effective teachers had higher amounts of time allocated to academics and higher pupil engagement than less effective teachers. High success rates on tasks were also found to be related to learning gains, with higher optimum success rates found for low-ability than for high-ability students.
Knapp (1995) studied 140 grade 1-6 classrooms in 15 high-poverty schools in California, Maryland, and Ohio over a two-year period. They found that students in grades 1, 3, and 5 who were exposed to meaning-oriented reading instruction performed 5.6 national curve equivalents (NCEs) higher, and students in grades 2, 4, and 6, 1.4 NCEs higher, at the end of the school year than students in classrooms with skills-oriented approaches to reading instruction. They also studied effects in math and writing and concluded that meaning-oriented instruction was effective in high-poverty classrooms. The teachers they observed teaching for meaning wanted to give children more responsibility for learning, wanted to provide academic tasks that asked more of students, and sustained engagement in learning among children.
The work of Wharton-MacDonald, Pressley, and Hampston (1998) both echoes and extends the earlier research on effective teachers of beginning reading. Three of the nine first-grade urban teachers in their sample were identified as most effective based on their students' end-of-year reading and writing achievement. These teachers demonstrated instructional balance, focusing on both literature and skills. They taught decoding skills explicitly and also provided their students with many opportunities to engage in authentic, integrated reading and writing activities. In contrast, the other teachers in the study either focused on skills or whole language approaches or combined the two in disjointed ways. The three most effective teachers extensively used scaffolding to help their students learn. They encouraged self-regulation by teaching their students to monitor their learning, the quality of their work, and their work time. They also encouraged self-regulation by teaching students to use strategies to be good readers and to fix problems they encountered as they were reading. The best teachers had high expectations for their students and masterful classroom management skills. They were skilled in managing time as well as behavior. They were well prepared for their lessons, and they mentioned the importance of routines in terms of activities and expectations. Finally, the most effective teachers were clear about the purposes of their activities and practices.
In the conclusion to their paper, Wharton-MacDonald et al. (1998) pointed to the need for additional research on the role of school factors and district policies on teacher practices and student performance. Hoffman (1991) also observed that there has been a paucity of research simultaneously investigating both school and classroom factors affecting reading achievement. Clearly, more research operating at the effective school-effective teacher nexus is needed. Such research would, in a single effort, examine school level factors (e.g., building climate, home-school relations, schoolwide organization for reading, collaborative efforts) while examining classroom/teacher factors (e.g., time spent in reading instruction, time on task, student engagement, approaches to word recognition and comprehension instruction, teachers' interactive styles).
In this study, we attempted to wed these important, but seldom integrated, lines of inquiry. We used quantitative and descriptive methods to examine the programs and practices in 11 moderate- to high-poverty schools selected because of their dual reputation for implementing recent reading reform and for beating the odds by promoting greater than expected primary-grade reading achievement. We also examined three schools chosen because they allegedly produced rather ordinary achievement. However, during the course of the data collection, some schools surfaced as more effective than others (see Stringfield et al., 1997, for a similar phenomenon). Therefore, rather than rely on a priori labels, we sought to pinpoint and explain school level (i.e., program) and classroom level (i.e., teachers' instructional practices) factors that distinguished the most effective schools from other schools in the study.
Method
Participants
Fourteen schools geographically dispersed throughout the country took part in the study, including schools in Virginia, Minnesota, Colorado, and California. A summary of the characteristics of each school, including type of intervention and type of schoolwide innovation, if any, in reading appears in Table 1. Schools in the study ranged from 28-92% poverty, and included four rural, four small town, and one suburban school, as well as five inner-city schools from three large metropolitan districts.
We started by trying to identify schools with two characteristics: (a) those that had recently implemented reform programs to improve reading achievement, and (b) those with a reputation for producing unexpectedly positive results with low-income populations. Because we were interested in special interventions for students most at risk for failure, we selected 8 schools which had carefully implemented an externally developed, research-proven early reading intervention, including 1 Book Buddies school (Invernizzi, Juel, & Rosemary, 1997), 2 Early Intervention in Reading schools (Taylor, Short, Frye, & Shearer, 1992), 3 schools with Right Start in Reading (Hiebert, Colt, Catto, & Gury, 1992), and 2 Reading Recovery schools (Pinnell, Lyons, DeFord, Bryk, & Seltzer, 1994). In 6 of these schools, the interventions were set within the broader context of program reform. The other 2 schools in this group of 8 had implemented early reading interventions without schoolwide reform of their reading program. Additionally, we selected two schools which had implemented externally developed, nationally recognized schoolwide reform programs--Success for All (Madden, Slavin, Karweit, Dolan, & Wasik, 1993) and Core Knowledge (Hirsch, 1987). Also included in the sample was one homegrown reform school; it had neither an externally developed schoolwide reform nor an externally developed early intervention. Operating on the assumption that all 11 of these reform schools might demonstrate similar achievement profiles, we included 3 typical comparison schools; these were schools with similar populations but with no history of either high achievement or reform activity. Two of the typical schools were in large urban districts and one was in a rural area. We wanted to include typical schools to provide a comparison base (both in terms of achievement and instructional practices) for the schools that had already undertaken and achieved some level of reform. All 3 of these schools were nominated by district administrators as typical rather than above average for the district in terms of primary-grade students' reading achievement.
Thus we began the study with 11 experimental schools and 3 control or comparison schools. However, as Stringfield et al. (1997) found in their work, not all schools believed to be exemplary in our study were, in fact, found to be so. Rather than rely on reputation, we decided to define school exemplarity empirically. We used a combination of gain scores from our own classroom reading measures plus scores on whatever achievement test the district normally used. Based upon this aggregate index, four schools in the present study were determined to be most effective and beating the odds. These schools were doing as well as or better than others in our sample in reading growth and/or doing better than average for their district, considering their poverty level. Six additional schools were determined to be moderately effective (neither exceptionally high nor low on the two indices that made up our school effectiveness rating), and four schools were determined to be least effective--lower than other schools in our study on our composite index, but typical for their district in primary-grade reading achievement. While we did not begin this research with an eye toward comparisons across three levels of effectiveness among the schools we selected, it turned out that the natural variations within our sample permitted us to meet our original goal of examining systematic relations among those performance outcomes, program elements, and instructional practices.
Within each building in each of grades K-3, the principal was asked to identify two teachers whom he or she felt were good or excellent teachers and who would be willing to take part in the study. We did not include all teachers because we wanted to focus on exemplary practice. Similar to what we discovered for schools, however, not all teachers were found to be exemplary, at least according to the judgments of our expert ratings of teacher accomplishment. They varied widely along the scale of accomplishment that we used to characterize their practices. The principals contacted the teachers they had nominated to request their participation. In three schools, principals felt there was only 1 kindergarten teacher who met the criteria for inclusion in the study. In another school, only 1 teacher in kindergarten and 1 in grade 2 were identified by the principal for inclusion in the study. In one school, first grade did not participate because it was involved in another research study. In those schools with an early intervention or other form of supplemental instruction, one resource teacher was selected by the principal. We started in the fall with 107 teachers; a total of 3 teachers from two different schools withdrew from participation shortly after the study began. Thus, a total of 22 kindergarten, 23 first-grade, 25 second-grade, 22 third-grade, and 12 resource teachers participated in the study. All teachers were female except for 2 male second-grade and 2 male third-grade teachers. The number of teachers in each school at each grade level is included in Table 1.
Each principal was also asked to participate in the study. The principal recruited the teachers, responded to a survey, completed an interview, and provided demographic information about the school, including the number of students on free and reduced lunch, the school's overall performance on district tests, and grade 3 standardized test achievement (expressed as a percentile).
Our goal was to collect pre- and posttest data for four children per classroom--two average performers and two low performers. Mindful of problems with permission, attrition, and absences, we asked each teacher, in the fall, to select four average-achieving and four low-achieving children, based on teachers' perceptions of reading performance (or emergent literacy performance), to complete pretests. If more than four children per classroom remained as we prepared to collect posttest data, we used achievement level and gender balance to reduce the classroom pool to four students.
Basic Data Collection Tools
Fall and spring outcome measures
Children were pretested in November and again in May. With the exception of the writing and spelling assessments, which were often administered by the classroom teachers in small-group settings, all tests were administered by members of our research team who had been specially trained for this project and the administration of these tests.
Kindergarten.
In the fall, children were individually tested on upper and lower case letter name identification (r = .96, alpha). In spring, the children were individually tested on upper and lower case letter name identification, rhyme (r = .88 alpha), phonemic blending and segmentation (r = .87, KR20, based on an earlier version of this test), and writing words. All kindergarten assessments came from the Emergent Literacy Survey (Pikulski, 1996), and none of the assessments was timed. For letter naming, the children were asked to name the upper case letters, presented in a scrambled order, and then name the lower case letters. For the phonemic segmentation and blending test, children were given an example of blending and an example of segmenting and then worked through six practice items. They were given the sounds in a word and asked to blend the sounds together into a word, or they were given a word and asked to give the sounds that were in the word. For the writing words test, they were asked to write as many words as they could. If needed, they were given the prompt, "Do you know your name, names of people in your family, names of animals, colors, any other words?" (Pikulski, 1996, p. 14).
Grade 1.
In the fall, children were individually tested on upper and lower case letter name identification, phonemic blending, phonemic segmentation (Pikulski, 1996), and a list of preprimer words (described below). Children were asked to try to read a preprimer passage from the Qualitative Reading Inventory II (QRI-II) (Leslie & Caldwell, 1995). Because 83% could not yet read the preprimer passage with at least 90% accuracy, it was dropped from our analysis.
In the spring, children were individually assessed on a specially constructed word reading test and reading passages from the QRI-II. However, the normal QRI procedure was modified to accommodate our interests. Starting with the primer passage, students' word recognition accuracy score was recorded. If they could read the primer passage with at least 90% accuracy, they went on to the grade 1 passage and continued moving upward until instructional level (90% accuracy or better) was found. If they could not read the primer passage they were asked to read the preprimer passage. Once an instructional level was determined, each student, irrespective of decoding ability, was asked to read a grade 1 passage so we could obtain a common measure for all students--the number of words a child could read correctly in one minute (wcpm). This became our common metric of fluency (Deno, 1985). The QRI was administered by our data collectors who received training and had written directions to follow in the administration of the QRI-II. Two members of our team scored one third of the children's fall QRI passage from which the wcpm measure was taken (interrater reliability = 97% agreement).
Also, children were asked to retell each passage they read. A 4-point holistic scoring rubric, developed by the St.Vrain Valley School District in Longmont, CO (Colt, 1997), was used to score the retellings on the passage that proved to be at their instructional level, 90% accuracy (the retelling rubric appears in Appendix A). All of the retellings were scored by a single member of our research team. A second member read and scored 15% of the retellings to establish interrater reliability (91% agreement).
The reading words test was developed by the research team for the project in order to ensure that our word test included an appropriate mix of decodable words. Half of the words at each grade level were high frequency words from the QRI-II and half were decodable words garnered from an extensive review of the decodable patterns introduced in the four most popular basal series. The decodable words were controlled to match the QRI-II words in terms of frequency (Carroll, Davies, & Richman, 1971). There were 20 words at each grade level, preprimer through grade 3, for a total of 100 words. A child started with the preprimer list and continued until she made seven consecutive errors. The reading words test is included in Appendix B. The total reading words score exhibited a fairly high correlation with the words correct per minute measure (r = .82 in grade 1, .71 in grade 2, and .69 in grade 3); its split half (odd-even) reliability was .97.
Grade 2.
In the fall, children were individually tested by a member of our research team on the reading words test for grade 1 and a grade 1 passage from the QRI-II. The number of words the child read correctly in the first minute was recorded. The child's word recognition accuracy on the passage was also recorded. Each child was asked to retell the passage, and the 4-point scoring rubric was used to score the retellings.
In the spring, children were individually tested by a member of the research team, starting with the reading words test. On the QRI-II, a procedure similar to the one used for grade 1 students was followed; in this case, each child began with a grade 1 passage and continued until an instructional level was found, after which each child read the grade 2 passage (to obtain the fluency measure) if it had not been read as a part of the procedure seeking to establish instructional level. For each passage, we asked the child for a retelling, and the retelling for the instructional level passage was scored using the St.Vrain 4-point rubric described earlier.
Grade 3.
In the fall, children were individually tested on reading words (grade 2 list) and on a grade 2 passage from the QRI-II. They were asked to read and retell the passage; from these data, three scores were computed: word recognition accuracy, fluency (wcpm), and retelling.
In the spring, third graders followed the same general procedure as second graders. They were individually tested on the reading words test; beginning with the preprimer list, they read successive lists until they missed seven consecutive words. They read passages from the QRI-II, starting with the grade 2 passage and continuing, either up or down, until an instructional level was found, at least through the grade 4 passage. For each passage read, a child's word recognition score and passage-retelling score was recorded. If it had not been encountered in the procedure to establish instructional level, every third grader was asked to read from the grade 3 passage so that a fluency measure (wcpm) could be obtained for every child on a grade-level passage.
Other measures
A writing sample based upon a common prompt and a graded spelling test, adapted from the work of Viisc (1994), were also administered in the fall for students in grades 2-3 and in the spring for all students in grades 1-3. Typically these were administered by the classroom teacher or another staff member at the school. An examination of these samples revealed such inconsistency in administration and such low levels of performance that they were dropped from any further analysis.
Observations
Members of our research team at each site were trained to conduct classroom observations. Observations occurred during an hour in which the teacher provided reading instruction. The observer recorded what the teacher was saying and doing as well as what the children were saying and doing during the lesson. Every five minutes the observer recorded any of the following teacher behaviors which were observed in the previous five-minute segment: coaching/scaffolding, modeling, engaging the children in recitation, explaining how to do something, telling, or engaging the children in a discussion. A description of each of these behaviors is provided in Table 2. Observers practiced coding video segments of instruction until they had at least 80% agreement with the principal investigators on the coding of teacher behaviors.
During the observations, the observers primarily focused on the teacher and on one low and one average target child (each of whom had been involved in the testing). However, comments on other children were included as well, and observers also provided comments about general classroom activity, involvement in the lesson, and other activities that seemed noteworthy. After the observation, the observer summarized the lesson by completing a summary form that required a statement about each of these characteristics: overall impression, teacher instruction and teacher-student interaction, activities and materials, student engagement, classroom management, and classroom environment.
Beginning in December, a member of our team conducted a one-hour observation of instruction during the basic reading program in every classroom once a month for five months. Observations were scheduled to accommodate each teacher's schedule. Complete observational data were obtained for 92 classroom teachers.
Logs
We asked every classroom teacher to keep a log of daily instructional activities in the classroom for one week in February and one week in April. We asked them to indicate how long they spent on various activities, including teacher directed reading of narrative and expository text; student independent reading; instruction in phonics, vocabulary, and comprehension; literature circles; writing in response to reading; other written composition; spelling; reading aloud to students; and other academic activities. Teachers recorded activities in 15-minute intervals and could include more than one activity during a time period. We divided the number of minutes for an interval by the number of activities coded to get number of minutes spent on an activity during that interval. For example, if a 15-minute interval was coded as whole-group phonics instruction, independent reading, and writing in response to reading, we coded each activity as occurring for a child for 5 minutes. Teachers also indicated the group setting in which each activity occurred: students working as a whole group, working in a small group, or working independently.
Using the raw data from the logs, we created three setting codes and six basic activity codes. The setting codes (e.g., large group, small group, and one-on-one) were fairly transparent and easy to translate into categories for further analysis. The original activity codes (e.g., phonics, comprehension, vocabulary, directed reading of narrative text, directed reading of informational text, literature circles) proved somewhat more challenging because of the ways in which teachers interpreted the directions. Thus we decided to create one large category labeled reading instruction (which included teacher-directed reading of narrative and expository text; instruction in phonics, vocabulary, and comprehension; and literature circles); the other original categories--independent reading, writing in response to reading, other written composition, spelling, reading aloud to students--remained intact. This data set allowed us to create a large number of different indicators of instructional emphasis. From those possibilities, presented in chart form in Table 3, we chose those variables in which the cells are marked with an X. In other words, we selected each of the activities summed over settings, each of the settings summed over activities, and a composite variable for total reading emphasis, which is the sum of the totals of columns 1-3 (reading instruction, independent reading, and writing in response to reading).
Questionnaires
In April or May, the principal and teachers from each school completed surveys which had been developed by a team of CIERA researchers for a broader national survey of beat the odds schools. The principal survey dealt with the following topics:
- Use of goals and standards;
- factors in improving reading/language arts instruction;
- approaches to professional development;
- planning reading/language arts instruction;
- attributes of effective schools;
- attributes of effective instruction;
- supplemental programs;
- role of special education teachers in reading/language arts instruction;
- perceptions of programs for struggling readers;
- increasing achievement for struggling readers;
- use of computers in reading/language arts;
- showcases for student work;
- consequences of assessment;
- assessment training; and
- reasons the school in question was successful in teaching students to read.
The teacher survey embraced most of the same topics and some additional ones:
- use of curriculum materials;
- aspirations for student reading levels tied to familiar trade books;
- time spent on reading/language arts activities;
- types, frequencies, and purposes of assessments;
- types of reading/language arts helpers and activities they engaged in;
- home-school communication and activities; and
- community activities.
Although the surveys were long (16 pages), the return rate was high. Across the 14 schools, 93 teacher surveys and 11 principal surveys were returned from the 104 teachers and 14 principals who participated in the study, for a return rate of 88%.
Interviews
We interviewed all principals and at least three teachers from each school. Principal questions focused on the community and links to parents, the principal's view of his or her leadership role, factors contributing to the school's success, challenges as well as things on which the school was still working, and advice to schools that wanted to significantly improve their reading achievement. Teacher questions were similar but also included questions about a teacher's general approach to teaching reading, behavior management systems, and her or his expectations for students. We transcribed the interviews and used them as a source of information for writing case studies for each school and for generating several school variables.
Case studies
For every school, a common outline was followed to guide our research team in crafting a case study for each school. A model case study was also provided to create some common expectations for content, format, and depth. Major topics within each case study included:
- School demographics;
- history of the school;
- primary-grade reading performance data;
- community/home/school relationships;
- school factors (leadership, effective schools/school change factors, professional development, format of school reading program);
- grade-level classroom practices (instruction, curriculum, student engagement, classroom management, classroom environment);
- other factors believed to be enhancing beginning reading achievement;
- challenges; and
- advice to other schools.
Case studies were written by the principal investigator or by the research team member who had spent the most time in a particular school. They ranged from 15-30 pages each.
Creating Variables for Data Analyses
Overall strategy
Quantitative and descriptive analyses were conducted using multiple sources of information used for both types of analysis. Analyses were conducted at the school and at the classroom level. As indicated, we made a decision fairly early on that we stood to learn the most from this initiative by examining the natural variability within our sample. As a general strategy, then, we developed empirically driven indices of effectiveness for both schools and teachers, grouped the schools or teachers into categories of effectiveness, and then examined the systematic differences among schools or teachers on other sets of variables. The procedures and criteria used to determine these levels will be described later. In addition, we developed categories and rating systems, built from data emerging from the observations, surveys, and interviews, that enabled us to operationalize both school-level and classroom-level variables. Tables 4a and 4b provide a summary of the school and teacher variables, including the procedures used to construct them.
Student outcomes
In order to examine the effects of the organizational and instructional factors that we observed in these schools, it was necessary to create outcome measures that reflected the effects that might conceivably have occurred during our year of observation. Therefore we decided to create residual scores for all the relevant spring reading measures, using appropriate fall scores as covariates. In grade 1, spring residual scores for reading fluency (wcpm) and retelling were created by using fall phonemic segmentation and blending scores as covariates. In grades 1-3, spring residual scores for reading words were created by using fall reading words scores as a covariate. In grades 2-3, spring residual scores for reading fluency and retellings were created by using fall reading fluency scores as a covariate. Within each grade level, these residual scores were converted to z-scores, which were calculated from the mean and standard deviation for the entire grade level sample, so that the data could be aggregated and/or compared across grade levels.
Constructing school variables
School effectiveness rating.
To be able to categorize schools as most effective, moderately effective, or least effective, we used two school-based measures: (a) students' growth on project measures of reading, and (b) students' performance on district measures of reading. These two scores were combined to create a general rating of effectiveness. First, the residual z-scores for retelling (at instructional level), fluency (wcpm on a grade level passage), and reading words were aggregated and standardized to create a composite index of reading growth. Second, we calculated what might be called a primary (as in primary grades) outcome index, using the end-of-grade-3 scores on the district-mandated test. As it turned out, these were, in each case, standardized achievement tests (six schools used the Stanford Achievement Test 9; two used the Metropolitan Achievement Test 7; two used the California Achievement Test; two used the Northwest Evaluation Association Levels Test; and two used a district-normed test.) A residual mean percentile score for each school was calculated by controlling for the school's poverty level (as indexed by the percentage of students receiving free or reduced lunch). This was done because students' achievement scores are depressed in schools with 50-75% of students living in poverty and seriously depressed in schools with 75-100% of students living in poverty (Puma et al., 1997).The residual scores were then converted to z-scores.
Third, the z-scores on the project measures and primary grades outcome measure were summed and standardized. When we examined these scores, we looked for natural breaks in the distribution that would divide the schools into three groups of approximately the same size. These breaks occurred at .5 standard deviations above and below the mean. Breaking at those two points yielded four most effective, six moderately effective, and four least effective schools (see Table 5). As the data in Table 5 suggest, the most effective schools were not necessarily populated by the most economically advantaged students.
|
Standard Composite Z-score From Z-scores (by Grade) for Residual WCPM 1 , Residual Retelling, Residual Reading Words |
Z-score for Residual Mean Percentile on District |
School Effectiveness Score-- |
School |
||
|---|---|---|---|---|---|
School efforts to link to parents.
An initial reading of the case studies indicated that schools varied in the extent to which they reached out to parents. Schools were judged to be high on the linking to parents factor if they had (a) a high mean score on teacher's level of home communication (described below under teacher factors), and (b) one or more of the following in place:
- An active site council in which parents helped to make school decisions;
- focus groups, phone surveys, or written surveys to find out parents' needs and concerns;
- family education programs on site in which meals and day care were provided; or
- a principal who called parents regularly to deliver good news about their children.
Schools were judged to be average on the linking to parents measure if they had one of the above factors (a or b) present in their school. Schools were judged to be low on the linking to parents measure if they demonstrated neither of the above factors or if they were low on the home communication teacher factor. Six schools were determined to be high on the linking to parents school factor, five were determined to be average on this factor, and three schools were determined to be low. The two raters evaluating schools on this factor achieved 93% agreement in their categorizations.
Systematic, internal assessment of pupil progress.
A school was coded as systematically assessing pupil progress if at least two thirds of teachers on the survey perceived this to be an attribute of their school, and if comments in the case studies and/or interviews supported this perception. Across schools, all but four were coded as systematically assessing pupil progress. These were not externally imposed standardized testing systems; they were internally developed systems for monitoring individual student progress within a schoolwide curriculum.
Building communication.
In a number of the interviews and surveys, staff commented about either good or poor collaboration and/or communication among teachers. Schools were judged to be high on the building communication scale if they had the following present: a high mean score on teachers' perceived building communication rating (from the teacher survey) and positive comments in the case study or in interviews about good collaboration among teachers within and across grades. Schools were judged to be average on this building communication factor if they had (a) an average mean score on the teachers' perceived building communication rating and no negative comments in the case study or interviews about building level communication or (b) a high mean on the teacher rating but negative comments about building level communication in the case study or interviews. A school was judged to be low on the building level communication factor if the teachers' mean building communication rating was low and negative comments about building level communication appeared in the case study or interviews. Two raters achieved 86% agreement on this judgment. Across the 14 schools in the project, 6 schools were judged to be high on the building communication factor, 5 were judged to be average, and 3 were judged to be low.
Use of an externally developed early reading intervention.
Within each grade level, a school was coded as either having an externally developed early intervention or no externally developed intervention in place. Out of the 14 schools, 10 had an externally developed intervention, 5 across two or more grades, 4 in grade 1 only, and 1 in grade 3 (in a grade 3-6 building.) Four schools had no externally-developed intervention in place.
Constructing classroom variables
Home communication.
On the survey, teachers indicated how often they communicated with parents and in what ways. The areas in which there appeared to be the most variability were the frequency with which teachers reported (a) calling home, (b) sending a letter or newsletter home, or (c) sending a traveling folder home. A 5-point scale was used to rate teachers on the extent to which they communicated with parents. (In the list below, letters, newsletters, and folders are all grouped together as artifacts.)
- (very high) Exceeds standards for level 4
- (high) Calls home at least once per month and sends home 2+ artifacts per week
- (moderate) Calls home once per month and sends home 2 artifacts per month or calls home 2-3 times per year and sends home 1 artifact per week
- (low) Calls home 2-3 times per year and sends home 1 artifact per month or calls home 1 time per year and sends home 1 artifact per week
- (very low) Does not meet standards for level 2
Across the sample of teachers in this project, 19% received a rating of 5, 19% a rating of 4, 40% a rating of 3, 14% a rating of 2, and 8% a rating of 1. The mean rating was 3.28 (SD = 1.16). One research team member rated all teachers, and a second rated a 25% sample; interrater agreement was 95%.
Student engagement.
From the observations, we searched for comments about children's engagement, which for us embraced both compliance (on-task behavior, in which children are productively engaged in their assigned activity) and involvement (evidence of genuine student enthusiasm for the activities). Most helpful in this regard were the summaries that each observer completed at the end of each observation session; one of their tasks was to summarize and point to evidence to support any conclusions about engagement. A teacher received a high rating (3) in maintaining student engagement if most of the comments on either of these topics (compliance or involvement) indicated that most students were engaged. A teacher was coded as average (2) if, in looking across comments about engagement, the comments were mixed--some indicating high student engagement and others indicating low engagement and/or off-task behavior. A teacher received a low (1) rating if numerous comments indicated that many students were off task. One member of our research team rated all teachers and another team member rated 25% of the teachers on this dimension. Across the pairs of ratings for each teacher, there was 100% agreement. Across teachers, the mean rating was 2.30 (SD = .76), with 48% of teachers receiving a rating of 3 (high), 34% a rating of 2 (average), and 18% a rating of 1 (low) in maintaining student engagement.
Preferred interaction style.
At the end of each five-minute segment during the classroom observations, observers coded instances of interactions observed during that segment, using these categories: coaching/scaffolding, modeling, engaging students in recitation, engaging students in discussion, explaining how to do something, or telling students information. (These activities are described in more detail in Table 3.) The total number of times a teacher was coded as engaging in each of these behaviors (more than one was possible within each segment) was calculated, and the behavior coded most frequently across all five observations was determined to be the teacher's preferred interaction style. Across teachers in grades 1, 2, and 3, 24% had a preferred interaction style of coaching, 31% had a preferred interaction style of engaging students in recitation, 39% had a preferred interaction style of telling students information, and 6% had a preferred style of modeling. To evaluate the trustworthiness of these ratings, three members of the research team rated a total of 25% of the teachers in terms of preferred interaction style based on a reading of the classroom observations and observation summaries for these teachers. There was 82% interrater agreement with the preferred interaction style as determined from the observation codes of interaction styles at five-minute intervals. If and when mismatches occurred, we reverted to code most frequently marked by the classroom observer.
Approaches to word recognition and comprehension instruction.
The data from the observations were analyzed to determine how teachers provided word recognition and comprehension instruction. A number of approaches were coded on a frequency scale, where each approach was determined to occur frequently, occasionally, or never during our observations for a particular teacher. The word recognition approaches included (a) coaching children in the use of strategies to figure out unknown words as they were reading text, (b) focusing on words in stories to review phonic elements, (c) providing explicit phonics instruction, and (d) practicing sight words.
Coaching involved prompting children to use a variety of strategies as they were engaged in reading during small-group instruction or one-on-one reading time. Typical prompts included phrases such as:
- What do you do when you come to a word you don't know?
- Does that make sense? (e.g., tease for teach)
- What do the letters ea say?
- How do you know it can't be dripped? (for dressed)
- Do you see a chunk you recognize? (in nothing)
- Let's sound it out and think of what would make sense
- Let's reread like good readers do.
Focusing on words in stories was realized most often as asking children to frame a word as they were reading in order to highlight a particular phonic element. For example, one teacher told the children to point to feet in the title of a new story and told the group that the ee said /e/ (long e sound). Another approach encouraged children to search for a particular element in their stories after reading. One teacher who was talking about -ed endings asked children to look for words in their story with -ed endings. Only four teachers in the entire sample were frequently observed using this approach, so it was eliminated from further analysis.
Explicit phonics instruction included work on a chart, whiteboard, worksheet, or word cards dealing with word study; word families; introducing or comparing phonic elements (i.e., er, ir, and ur all have the same sound); making words (Cunningham & Cunningham, 1992); writing words; and reading words with a particular phonic element in isolation. Practice on sight words involved teachers using flash cards, a pocket chart, or a word wall to review words the students were expected to recognize instantly as sight words.
For comprehension instruction, eight different instructional practices were observed and coded: doing a picture walk; asking for a prediction; asking a text-based question; asking a higher level, aesthetic response question; asking children to write in response to reading (including writing answers to questions about what they had read); doing a story map; asking children to retell a story; and working on a comprehension skill or strategy. Each grade 1-3 teacher was coded as frequently, occasionally, or never observed engaging in each of the above practices. For further analysis, we focused on those categories for which 10 or more teachers were frequently observed to have used the strategy: asking text-based questions, asking higher level questions, and asking children to write in response to what they had read.
One member of the research team coded all teachers and a second member coded 25% of the teachers. There was 100% agreement on coding of coaching for word recognition during reading, 94% agreement on providing explicit phonics instruction, 94% agreement on practicing sight words, 96% agreement on asking text-based questions, 96% agreement on asking higher level questions, and 100% agreement on writing in response to reading.
Teacher accomplishment rating.
Two experts in teacher supervision at the elementary school level, both members of our research team, read all teacher observations. One expert was a whole language advocate, and the other described herself as a reading skills advocate. They used a checklist of elements of effective instruction from the principal survey (based on Anderson et al., 1979; Barr & Dreeban, 1991; Hoffman, 1991; Pressley, Rankin, & Yokoi, 1996; Roehler & Duffy, 1991; Wharton-MacDonald et al., 1998) and a checklist of elements of culturally responsive teaching from the work of Ladson-Billings (1994; see Table 6). Although the two lists have a fair amount of overlap, they also tap different aspects of skillful teaching.
For each item in each list the raters estimated, based on a thorough reading of the observations and summaries, the quantity of indicators of demonstrated accomplishment by a teacher: many, some, few, or could not determine. Each rater rated each teacher separately on the two scales. For each teacher, the raters examined each item on each scale separately, providing a judgment about whether there were many, some, or few instances of that attribute in the data set (or if there were data, they coded the item as could not determine. In 98% of the cases, a teacher received the same score on both scales. The two raters then arrived at a composite rating (on a 3-point scale) based upon a mean of the two scores. For the overall rating, the two raters demonstrated 80% agreement on their overall ratings. The two raters had used a system of pluses and minuses along with their numerical ratings. When ratings less than a point apart (e.g., 2-, 1+; 3, 2+) were considered, the two raters had 94% agreement. There were only six instances in which the raters disagreed by a point, and these disagreements were moderated by consultation with the senior author of this paper. Across teachers in our sample, 41% were identified as demonstrating many of these elements (most accomplished), 32% demonstrated some of these elements (moderately accomplished), and 27% demonstrated few of the elements of culturally responsive teaching and effective instruction (least accomplished).
Results and Discussion
The results are organized by level of analysis. First we report and discuss results, largely descriptive, at the school level. Second, we examine the variations among instructional practices of teachers within the levels of school effectiveness that emerge from the school level analysis. Third, we report and discuss the variations we found in instructional practice as a function of levels of teacher accomplishment (how they were rated on the two-pronged scale--effective teaching and culturally responsive pedagogy). Finally, we report on a regression analysis combining school and teacher variables. Throughout the results we refer to the most effective schools by name: Hilltop, Stevenson, Wheeler, and Woodlawn (all pseudonyms).
Due to the complexity of the study, the fact that many of the classroom variables focus on grades 1-3 (e.g., student level of engagement, time spent in small- or whole-group instruction, preferred interaction style), and the use of different outcome measures, the kindergarten classrooms were dropped from the analysis. A brief summary of findings are in Appendix C.
|
N 3 |
||||||||
|---|---|---|---|---|---|---|---|---|
|
WCPM 4 on Grade 1 Passage |
||||||||
|
School |
||||||||
|
Teacher rating 6 |
||||||||
The unit of analysis depended on the level. Some analyses involving school factors were based on the school-level scores (e.g., school-level ratings, composite school effectiveness scores that had been summed across grade level) for each of the 14 schools. Most analyses at the school and all at the teacher level were based on classroom mean scores. Mean classroom performance measures by school effectiveness and teacher accomplishment levels are reported, for archival purposes, in Tables 7-9.
School Level Analyses
Linking to parents
Parent links were positively and statistically significantly related (r = .73) to the school effectiveness rating and to all measures of student growth, fluency (r = .60), retelling (r = .37), and reading words (r = .41; see Table 10).
|
N 7 |
|||||||
|---|---|---|---|---|---|---|---|
|
WCPM 8 |
|||||||
|
School |
|||||||
|
Teacher rating 10 |
|||||||
The most effective schools reported more links with parents than the moderately effective and least effective schools. Three of the four most effective schools reported having an active site council in which parents served on the committee with teachers and other school staff and helped to make decisions concerning school practices. Four of six moderately successful schools also reported having an active site council, but only one of the four least successful schools reported having such a body in place.
|
N 11 |
|||||||
|---|---|---|---|---|---|---|---|