Evaluation of the Michigan Literacy Progress Profile (MLPP)

 

 

Robert D. Carpenter

University of Michigan

Center for the Improvement of Early Reading Achievement (CIERA)

 

 

 

 

Contact information:

e-mail: phaedrus@umich.edu

CIERA: www.ciera.org
Evaluation Team

 

 

Michigan State University          University of Michigan

 

Dr. P. David Pearson                                 Dr. Scott Paris, Professor

Distinguished Professor of Education          CIERA, Department of Psychology,

Combined Program in Education and Psychology

        

Gina Cervetti                                            Robert Carpenter

         Sharman Siebenthal                                   Alison Paris

         Brigette Laier                                           Graduate Researchers

Graduate Researchers                                 CIERA, Combined Program in                                                                                            Education and Psychology

 

Denise Larabell

Research Assistant

 

 

 

Ingham Intermediate School District

 

Tamara Bashore-Berg, Supervisor

Department of Planning and Evaluation

 

Kathy Humphrey, Evaluation Specialist

Department of Planning and Evaluation

 

Jonathan Flukes, Evaluation Assistant

Department of Planning and Evaluation

 

Kelly Trout, Evaluation Assistant

Department of Planning and Evaluation

 

Allen Strozewski, Computer Technician

Department of Planning and Evaluation

 

Chris Quinn, Secretary

         Department of Planning and Evaluation


Overview

 

 

Š   Overall Evaluation

 

Š   Year 1 Test/Retest Reliability

 

Š   Year 1 Concurrent Validity

 

Š   Year 1 Summary

 

Š   Year 2 Overview

 


Overall Evaluation

 

Primary Goal:

 

To evaluate the reliability, validity, and utility of the MLPP

 

 

Evaluation Timeline

 

–Year 1

•Test/Retest Reliability

•Concurrent Validity

 

–Year 2

•Predictive Validity

•Consequential Validity

 


Purposes of Year 1

 

 

Š   To collect baseline data for long-term predictive validity of the MLPP

 

Š   Examine the test/retest reliability of the MLPP enabling skills

 

Š   Examine the test/retest reliability of the MLPP oral reading assessments

 

Š   To test the concurrent validity of both the enabling and oral reading tasks

 


Method

 

 

Š   Data were collected in the spring and fall of 2001

 

Š   For test/retest reliability students were tested on the same tasks 2-4 weeks apart by teachers and researchers

 

Š   Concurrent validity established using teacher and researcher MLPP data and researcher collected TPRI and Gates data

 


Sample

 

Š   Four sites across the state were selected because they represented rural, urban, and suburban communities and were strong implementors of the MLPP

 

Š   More than 700 K-3 students approximately 25% in each grade

 

Š   41 % Female

 

Š   68 % Caucasian

 


UM Specific Procedures

 

 

 

Š   Met administrators and teachers to explain project in spring 2001

 

Š   Retested students using the same MLPP assessments and procedures as teachers 2-4 weeks after the teacher administration (spring 2001)

 

Š   Tested students using the TPRI (spring 2001) and Gates (fall 2001) at Milan during a similar 4-week window

 


MLPP Enabling Components

 

 

Š   Letter Identification

 

Š   Letter-Sound Identification

 

Š   Hearing and Recording Sounds

 

Š   Phonological Awareness

–Rhyme

–Onset Rim and Blending

–Segmentation

 

Š   Concepts of Print

 

Š   Known Words

MLPP Oral Reading Framework

 

 

 

Š   Sight Word/Decodable Word List

 

Š   Oral Reading

–Accuracy

–Fluency

–Retelling

–Comprehension

 

 


Test/Retest Correlations for MLPP Enabling Tasks

 

Task

Milan

Willow Run

Letter ID

.94***

.97***

Letter Sound ID

.91***

.74***

Phonemic Awareness

 

 

      Rhyming

.47*

.73***

      Blending

.78***

.91***

      Segmentation

.10

.68***

H and R Sounds

 

 

.93***

Concepts of Print

.65***

.33

Known Words

 

.73***

 

* = p< .05; ** = p< .01; *** = p< .001


Letter Identification

 

Milan Scatterplot of Teacher and Researcher Scores

Overall Regression Line

 

Agreement Line  

 
 

 

Willow Run Scatterplot of Teacher and Researcher Scores

Regression Line

 

Agreement Line  

 


Letter Sound ID

 

Milan Scatterplot of Teacher and Researcher Scores

Overall Regression Line

 

Agreement Line  

 

 

Willow Run Scatterplot of Teacher and Researcher Scores

Regression Line

 

Agreement Line  

 


Phonemic Awareness- Rhyming

 

Milan Scatterplot of Teacher and Researcher Scores First Grade Only

Regression Line

 

Agreement Line  

 

 

Willow Run Scatterplot of Teacher and Researcher Scores

Overall Regression Line

 

Agreement Line  

 
 


Phonemic Awareness- Blending

 

Milan Scatterplot of Teacher and Researcher Scores

Regression Line

 

Agreement Line  

 

 

Willow Run Scatterplot of Teacher and Researcher Scores

Regression Line

 

Agreement Line  

 
 


Phonemic Awareness- Segmentation

 

Milan Scatterplot of Teacher and Researcher Scores

Regression Line

 

Agreement Line  

 

 

Willow Run Scatterplot of Teacher and Researcher Scores

Regression Line

 

Agreement Line  

 


Hearing and Recording Sounds

 

Willow Run Scatterplot of Teacher and Researcher Scores

Regression Line

 

Agreement Line  

 
 


Concepts of Print

 

 Milan Scatterplot of Teacher and Researcher Scores

Overall Regression Line

 

Agreement Line  

 
 

Willow Run Scatterplot of Teacher and Researcher Scores

Regression Line

 

Agreement Line  

 
 


Known Words

 

Willow Run Scatterplot of Teacher and Researcher Scores

 

Regression Line

 

Agreement Line  

 

 

 

 


Test/Retest Correlations for MLPP Oral Reading

 

Task

Milan

Willow Run

Word Recognition (Highest List)

 

.66***

.78***

Word Recognition (Scale)

 

.94***

.98***

Accuracy

.51***

.46***

Fluency

.54**

 

Rate

.78***

 

Comprehension

.35*

 

Retelling

.53***

.10

 

* = p< .05; ** = p< .01; *** = p< .001   


Highest Sight Word List

 

Milan Scatterplot of Teacher and Researcher Scores

Overall Regression Line

 

Agreement Line  

 

 

Willow Run Scatterplot of Teacher and Researcher Scores

Overall Regression Line

 

Agreement Line  

 


Total Sight Word Score

 

Figure 19. Milan Scatterplot of Teacher and Researcher Scores

Overall Regression Line

 

Agreement Line  

 

 

Figure 20. Willow Run Scatterplot of Teacher and Researcher Scores

Overall Regression Line

 

Agreement Line  

 


Accuracy

 

Milan Scatterplot of Teacher and Researcher Scores

Overall Regression Line

 

Agreement Line  

 

 

Willow Run Scatterplot of Teacher and Researcher Scores in Decimals

Overall Regression Line

 

Agreement Line  

 
 

Fluency

 

 

Highest Passage Scatterplot of Teacher and Researcher Scores

Overall Regression Line

 

Agreement Line  

 


Reading Rate

 

 

Scatterplot of Highest Passage Reading Times

Overall Regression Line

 

Agreement Line  

 

 


 

Comprehension

 

 

Scatterplot for Highest Passage

Overall Regression Line

 

Agreement Line  

 
 


Retelling

 

Milan Scatterplot of Highest Passage Percent Propositions

Overall Regression Line

 

Agreement Line  

 

 

Willow Run Scatterplot of Retelling Rubric Scores

Overall Regression Line

 

Agreement Line  

 


 

Concurrent Validity

for MLPP Tasks

 

 

Task

TPRI

GMRT

Letter ID

 

.94***

.82**

Phonemic Awareness

 

.19

      Rhyming

.43**

 

      Blending

.81***

 

Word Recognition

.79***

 

Accuracy

.41*

 

Comprehension

.07

.71***

 

* = p< .05; ** = p< .01; *** = p< .001


Summary of

Year 1 Findings

 

 

Š   High reliability for enabling skills

 

Š   Moderate reliability for oral reading tasks

 

Š   Moderate concurrent validity with the TPRI and the GMRT

 


Year 2

 

 

Š   We are currently collecting MEAP data on 3rd grade students from year one in addition to the MEAP data of participants in previous MLPP studies to measure predictive validity

 

Š   Spring 2002 we will collect surveys and interviews of students, teachers and administrators to determine the consequential validity of the MLPP