Combining Generalizability Theory with Item Response Modeling
Generalizability theory can provide very useful information for guiding assessment design and deployment, in terms of controlling error variance. Item response modeling can provide a very useful means of analyzing and disseminating information about student performances from assessments. But the mathematical models underlying the two methods are based on fundamentally different assumptions, which makes it difficult to take advantage of the advantages of both (i.e., most often reporting is done on the basis of just one method).
Capitalizing on a recent Bayesian formulation of item response modeling (Patz, et al, in press), we plan
- to compare the "standard implementations" of the two methods,
- to develop, as far as we can, techniques within each that match the techniques of the other,
- to develop an overarching framework that will encompass the essential elements of both, and
- to implement that framework for some example assessments in science.
REFERENCES:
Patz, R. J., Junker, B. W., Johnson, M. S., & Mariano, L. T. (in press). The hierarchical rater model for rated test items and its application to large-scale assessment data. Journal of Educational and Behavioral Statistics.
Researchers: Mark Wilson, UC Berkeley
