NEWS FROM THE PROJECT
What is Cultural Validity in Assessment?
What is Cultural Validity in Assessment?by Guillermo Solano-Flores
Summer, 2000
Culture influences how we behave, what we believe, what we value, how we socialize, and how we make sense of our experiences. Lack of awareness that different cultures promote different behaviors, beliefs, values, and styles of socializing and learning can jeopardize understanding among individuals. For example, in some cultures children learn to show respect to adults by not looking them in the eye. An adult from a different culture who is not aware of this subtle difference may perceive a child who exhibits this behavior as disrespectful.
Lack of awareness of cultural differences may also affect the way in which we judge student academic performance. For example, students from certain cultures may tend to participate little or to give very short responses when they are asked questions in class, not necessarily because they have poor command of a topic, but because in their cultures long verbal interactions are discouraged among children and showing how much one knows is considered inappropriate. Due to these cultural differences, these students might be incorrectly perceived as shy or nonverbal, and their academic performance might be underestimated.
Ignoring the importance of cultural differences may also lead to unfair testing practices. For example, the wording, illustrations, layout and contextual information in a test item may reflect the language, ways of thinking and experience of a particular cultural group. In this case, the item might privilege students from that same cultural group and penalize students from other cultural groups.
The potential consequences of unfair testing practices is especially relevant in the United States, where cultural diversity is considerable, and where decisions that affect students’ lives and the funding received by schools are based on standardized test scores. Tests that are developed without properly taking into account cultural differences may be reinforcing social inequalities.
Methods intended to detect and reduce test cultural bias have existed for decades. Although well-intentioned and necessary, these methods fail to consider the ways in which culture influences thinking as students engage in assessments. They are almost exclusively based on comparing the performance of different cultural groups by means of statistical analyses.
The notion of cultural validity in assessment addresses the fact that society and culture may influence the way in which individuals interpret test items and respond to them. A test cannot be considered as culturally valid if these sociocultural influences are not taken into account throughout the entire process of its development. Assessing the Cultural Validity of Science and Mathematics Assessments (which project staff have dubbed Cultural Validity in Assessment, or CVA) is funded by the National Science Foundation. Its ultimate goal is to contribute to attaining equitable testing in science and mathematics by offering a new perspective for addressing cultural diversity in testing.
Ten cultural groups are participating in this study. We have selected a sample of science and mathematics exercises that have been used in the past in large-scale testing. We are giving these exercises to students and asking them to “think aloud” as they solve the problems.
Based on these verbalizations and on interviews with students, we intend to determine whether different cultural groups exhibit different patterns in understanding science and mathematics exercises as well as how culture influences the thinking elicited by those exercises.
If our research shows that culture influences the way in which students interpret and solve science and mathematics exercises, and that not considering cultural differences may penalize students from some cultural groups, it will provide evidence that cultural validity is a form of test validity that should be considered systematically in assessment development and testing practices.
Suggested Readings
Kopriva, R. & Sexton, U. M. (1999). Guide to Scoring LEP Student Responses to Open-Ended Science Items. Washington, D.C.: Council of Chief State School Officers.
Kusimo, P., Ritter, M.G., Busick, K., Ferguson, C., Trumbull, E., & Solano-Flores, G. (2000). Making assessment work for everyone: How to build on student strengths. Regional Educational Laboratories.
Oakes, J. (1990). Multiplying inequalities: The effects of race, social, class, and tracking on opportunities to learn mathematics and science. Santa Monica, California: The RAND Corporation.
Solano-Flores, G. & Nelson-Barber, S. (2000). Cultural validity of assessments and assessment development procedures. Paper presented at the annual meeting of the American Educational Research Association. New Orleans, LA, April.
Cultural Validity in Assessment: An Update
by Guillermo Solano-Flores
Summer, 2001
Our project, Assessing the Cultural Validity of Science and Mathematics Assessments, ultimately seeks to contribute to attaining equitable testing. We intend to determine whether thinking, communication, and learning styles inherent to culture influence how students interpret and respond to science and mathematics tests. We hope to learn how culture should be taken into account to produce fair tests. Using the notion of cultural validity to examine the quality of items from existing large-scale tests, we intend to see how students from different cultural backgrounds make sense of those items and how they relate their content to their everyday lives.
In our study, students complete a short test with two science and two mathematics items and a questionnaire on activities they engage in at school and outside school and which may be related to the content of the items. We also interview some students individually and ask them to describe how they relate their everyday personal experiences to the content of the items. Since September of 2000, we have visited schools in Saipan (Commonwealth of the Northern Mariana Islands), Washington, Alaska, California, New York, and Washington D.C. in urban, sub-urban or rural settings Participating students constitute a good sample of the rich ethnic mosaic of this country: Chamorros and Carolinians from the Pacific, Hispanics, Yupíiks (Eskimos), Asians, African-Americans, Anglo-Saxons, and Haitians.
What do we expect to learn from this study? We hope to identify patterns in which culture and cognitive activities interact for students of the same cultural background. We want to determine whether students from different cultural groups exhibit different patterns in which they understand science and mathematics exercises and how those differences can account for performance score differences between cultural groups.
The implications of the study can be serious, as it may shed new light on how science and mathematics test items should be developed in a way that honors cultural diversity while measuring the same high standards desired for all students .We may learn, for instance, that new and improved methods for developing assessments should be created which would address more effectively the way in which items must be worded .These new methods would ensure that the imaginary situations and stories used with the intent to make an item meaningful do not lay on flaky assumptions about student’s experiences, lives and values. From April through June, after collecting data from ten cultural groups, we will meet with elementary school teachers. They will score the students responses from the written test and will code the responses to questionnaires and interviews. All materials and students’ names will be kept confidential.
During the summer, we will analyze the data, and will devote the last three months of the year to reporting our results to the National Science Foundation, our funding agency. We will also disseminate our results among teachers, school administrators, parents, educators, and researchers. During that final stage of the project, we will translate the results from our investigation into concrete actions for schools, and produce formal recommendations for decision and policy makers. What does our study indicate thus far? Our preliminary observations reinforce the notion that personal experience strongly influences how students interpret items and respond to them .Students may use knowledge acquired through formal instructional experiences at school. However, the informal, first-hand experiences they have at home and within their communities shape the way they make sense of test items. Our preliminary observations also suggest that the way in which some science and math items are worded may in certain cases not work in favor of some students. That may be the case even if English proficiency is not an issue. For reasons yet to be determined, some students demonstrate different levels of competence in science and math depending on whether they are tested in a paper-and-pencil format or interviewed informally.
Because they are preliminary, these observations should be taken with caution. We cannot draw any formal conclusions until we reach the completion of our project. However, the results thus far speak to the relevance of culture as a factor that must be considered in the development of science and mathematics tests.
Suggested Readings
Kusimo, P., Ritter, M.G., Busick, K., Ferguson, C., Trumbull, E., & Solano-Flores, G. (2000). Making assessment work for everyone: How to build on student strengths. San Francisco, CA: Regional Educational Laboratories.
Smith, G. Pritchy. (1998). Common sense about uncommon knowledge: The knowledge bases for diversity. Washington, DC: AACTE Publications.