 |
|
In analyzing a school’s scores, year-to-year change analysis tends to be highly unreliable. Year-to-year changes in student achievement could be due to chance rather than to the impact of changes in school programs and practices. In addition, schools that are truly making gains in student achievement have a process for making continuous, incremental improvement over time – increase in student achievement is not simply a one-year phenomenon. For these reasons, a multi-year analysis, the quasi-cohort method, was used to reliably track student achievement gains over time.
The quasi-cohort method is used if the school meets all of the following criteria:
- The school needs to examine trends across years.
- The school has a single assessment instrument that is administered each year to the whole school.
- The school has grade level summary data available from this school-wide assessment.
This method compares the performance of the "same group of students" as they progress through grade levels year after year. For example, the year 2000 results for students in grades 9 will be compared to the year 2001 results for students in grades 10. In this analysis, the grade 9 students and the grade 10 students are viewed as the "same group." Ideally, the quasi-cohort method is an analysis of "matched students" over time. Thus this method is not as effective when a significant number of students are retained or when there is a significant amount of mobility from one year to the next.
The following is a process schools can use to determine whether there is a positive or negative gain in student achievement (refer to the Table (PDF format)):
- Enter into the table the percentages of students at/above proficiency at each grade level.
- Determine which data to compare (i.e., year 2000 results for grades 9 & 10 with year 2001 results for grades 10 & 11).
- Calculate the difference between the two years (i.e., year 2001 results minus year 2000 results).
The year-to-year changes can be summed for an overall single measure across many years, but the pattern across years may be more meaningful to make inferences about continuous change.
The Tables (PDF format) for the quasi-cohort method use different colors to help the reader find the appropriate data to compare. The appropriate school year-end results and the annual change result have the same color.
The following methods are multi-year analyses with which schools may be familiar. However, as noted below, it is important for schools to understand the specific uses and limitations of each method.
Longitudinal Analysis
This analysis compares results from a baseline year to a point several years later, for example, comparing results from 1998 to results from 2001. This method does not expose “ups and downs” within the span of time and runs the risk of including only a small fraction of the student population, if there is high student mobility at the school.
If using this method, it is recommended that schools look for trends of consistent or similar annual gains by comparing results from 1998 to 1999, 1999 to 2000, 2000 to 2001, and so on.
Matched Scores Analysis
This analysis compares the results of the same students year after year. In order to carry out this type of analysis, individual student test scores, each with a unique student identification number must be kept in a database so that test results for a given student can be matched and compared across years. This type of analysis is one of the most accurate and reliable for examining gains or looses on a single assessment instrument; however, only students with test scores for more than one consecutive years can be included in the analysis. An additional limitation is that schools do not often have individual student data available for this type of analysis.
Whole School Scores
This analysis uses a complex, weighted formula to yield a score for a whole school. California’s Academic Performance Index (API) is an example of a whole school score comparison. Statistical studies on the whole school method indicate that this method is not reliable for the purpose of looking at grade level trends over time.
|
|
|
|
