Evaluation of the Michigan Literacy
Progress Profile (MLPP)
Robert D. Carpenter
University of Michigan
Center for the Improvement of Early
Reading Achievement (CIERA)
Contact information:
e-mail: phaedrus@umich.edu
CIERA: www.ciera.org
Evaluation Team
Michigan State University University
of Michigan
Dr. P. David Pearson Dr.
Scott Paris, Professor
Distinguished
Professor of Education CIERA,
Department of Psychology,
Combined Program in Education and Psychology
Gina Cervetti Robert
Carpenter
Sharman
Siebenthal Alison
Paris
Brigette
Laier Graduate
Researchers
Graduate Researchers CIERA,
Combined Program in Education
and Psychology
Denise Larabell
Research Assistant
Ingham
Intermediate School District
Tamara Bashore-Berg,
Supervisor
Department of Planning and
Evaluation
Kathy Humphrey, Evaluation
Specialist
Department of Planning and
Evaluation
Jonathan Flukes, Evaluation
Assistant
Department of Planning and
Evaluation
Kelly Trout, Evaluation
Assistant
Department of Planning and
Evaluation
Allen Strozewski, Computer
Technician
Department of Planning and
Evaluation
Chris Quinn, Secretary
Department of
Planning and Evaluation
Overview
á Overall Evaluation
á Year 1 Test/Retest Reliability
á Year 1 Concurrent Validity
á Year 1 Summary
á Year 2 Overview
Overall Evaluation
Primary
Goal:
To evaluate the reliability, validity, and utility
of the MLPP
Evaluation
Timeline
ÐYear 1
¥Test/Retest Reliability
¥Concurrent Validity
ÐYear 2
¥Predictive Validity
¥Consequential Validity
Purposes of Year 1
á To
collect baseline data for long-term predictive validity of the MLPP
á
Examine the
test/retest reliability of the MLPP enabling skills
á
Examine the
test/retest reliability of the MLPP oral reading assessments
á
To test the
concurrent validity of both the enabling and oral reading tasks
Method
á
Data were collected in the spring and fall of 2001
á
For test/retest reliability students were tested on the same tasks
2-4 weeks apart by teachers and researchers
á
Concurrent validity established using teacher and researcher MLPP
data and researcher collected TPRI and Gates data
Sample
á
Four sites across the state were selected because they represented
rural, urban, and suburban communities and were strong implementors of the MLPP
á
More than 700 K-3 students approximately 25% in each grade
á
41 % Female
á
68 % Caucasian
UM Specific Procedures
á
Met administrators and teachers to explain project in spring 2001
á
Retested students using the same MLPP assessments and procedures
as teachers 2-4 weeks after the teacher administration (spring 2001)
á
Tested students using the TPRI (spring 2001) and Gates (fall 2001)
at Milan during a similar 4-week window
MLPP Enabling Components
á Letter Identification
á
Letter-Sound Identification
á Hearing
and Recording Sounds
á
Phonological Awareness
ÐRhyme
ÐOnset Rim and Blending
ÐSegmentation
á
Concepts of Print
á Known
Words
MLPP Oral Reading Framework
á
Sight Word/Decodable Word List
á
Oral Reading
ÐAccuracy
ÐFluency
ÐRetelling
ÐComprehension
Test/Retest Correlations for MLPP Enabling Tasks
|
Task
|
Milan
|
Willow
Run
|
|
Letter ID
|
.94***
|
.97***
|
|
Letter
Sound ID
|
.91***
|
.74***
|
|
Phonemic Awareness
|
|
|
|
Rhyming
|
.47*
|
.73***
|
|
Blending
|
.78***
|
.91***
|
|
Segmentation
|
.10
|
.68***
|
|
H and R Sounds
|
|
.93***
|
|
Concepts
of Print
|
.65***
|
.33
|
|
Known
Words
|
|
.73***
|
* = p<
.05; ** = p< .01; *** = p< .001
Letter Identification
Milan Scatterplot of Teacher and Researcher Scores
Overall Regression Line
Agreement Line  
|
|
Willow Run Scatterplot of Teacher and Researcher Scores
Regression Line
Agreement Line  
|
|
Letter Sound ID
Milan Scatterplot of Teacher and Researcher Scores
Overall Regression Line
Agreement Line  
|
|
Willow Run Scatterplot of Teacher and Researcher Scores
Regression Line
Agreement Line  
|
|
Phonemic Awareness- Rhyming
Milan Scatterplot of Teacher and Researcher Scores First
Grade Only
Regression Line
Agreement Line  
|
|
Willow
Run Scatterplot of Teacher and Researcher Scores
Overall Regression Line
Agreement Line  
|
|
Phonemic Awareness- Blending
Milan Scatterplot of Teacher and Researcher Scores
Regression Line
Agreement Line  
|
|
Willow Run Scatterplot of Teacher and Researcher Scores
Regression Line
Agreement Line  
|
|
Phonemic Awareness- Segmentation
Milan Scatterplot of Teacher and Researcher Scores
Regression Line
Agreement Line  
|
|
Willow Run Scatterplot of Teacher and Researcher Scores
Regression Line
Agreement Line  
|
|
Hearing and Recording Sounds
Willow Run Scatterplot of Teacher and Researcher Scores
Regression Line
Agreement Line  
|
|
Concepts of Print
Milan
Scatterplot of Teacher and Researcher Scores
Overall Regression Line
Agreement Line  
|
|
Willow
Run Scatterplot of Teacher and Researcher Scores
Regression Line
Agreement Line  
|
|
Known Words
Willow Run Scatterplot of Teacher and Researcher Scores
Regression Line
Agreement Line  
|
|
Test/Retest Correlations for MLPP Oral Reading
|
Task
|
Milan
|
Willow
Run
|
|
Word Recognition (Highest List)
|
.66***
|
.78***
|
|
Word Recognition (Scale)
|
.94***
|
.98***
|
|
Accuracy
|
.51***
|
.46***
|
|
Fluency
|
.54**
|
|
|
Rate
|
.78***
|
|
|
Comprehension
|
.35*
|
|
|
Retelling
|
.53***
|
.10
|
* = p< .05; ** = p<
.01; *** = p< .001
Highest Sight Word List
Milan Scatterplot of Teacher and Researcher Scores
Overall Regression Line
Agreement Line  
|
|
Willow Run Scatterplot of Teacher and Researcher Scores
Overall Regression Line
Agreement Line  
|
|
Total Sight Word Score
Figure 19. Milan Scatterplot of Teacher and
Researcher Scores
Overall Regression Line
Agreement Line  
|
|
Figure 20. Willow Run Scatterplot of Teacher and
Researcher Scores
Overall Regression Line
Agreement Line  
|
|
Accuracy
Milan Scatterplot of Teacher and Researcher Scores
Overall Regression Line
Agreement Line  
|
|
Willow Run Scatterplot of Teacher and Researcher Scores in
Decimals
Overall Regression Line
Agreement Line  
|
|
Fluency
Highest Passage Scatterplot of Teacher and Researcher Scores
Overall Regression Line
Agreement Line  
|
|
Reading Rate
Scatterplot of Highest Passage Reading Times
Overall Regression Line
Agreement Line  
|
|
Comprehension
Scatterplot for Highest Passage
Overall Regression Line
Agreement Line  
|
|
Retelling
Milan Scatterplot of Highest Passage Percent Propositions
Overall Regression Line
Agreement Line  
|
|
Willow Run Scatterplot of Retelling Rubric Scores
Overall Regression Line
Agreement Line  
|
|
Concurrent Validity
for MLPP Tasks
|
Task
|
TPRI
|
GMRT
|
|
Letter ID
|
.94***
|
.82**
|
|
Phonemic
Awareness
|
|
.19
|
|
Rhyming
|
.43**
|
|
|
Blending
|
.81***
|
|
|
Word
Recognition
|
.79***
|
|
|
Accuracy
|
.41*
|
|
|
Comprehension
|
.07
|
.71***
|
* = p< .05; ** = p< .01; *** = p< .001
Summary
of
Year
1 Findings
á High reliability for enabling skills
á Moderate reliability for oral reading tasks
á Moderate concurrent validity with the TPRI and the GMRT
Year
2
á We are currently collecting MEAP data on 3rd
grade students from year one in addition to the MEAP data of participants in
previous MLPP studies to measure predictive validity
á Spring 2002 we will collect surveys and interviews of
students, teachers and administrators to determine the consequential validity
of the MLPP