Cultural Validity:
The Need for a Socio-Cultural Perspective in Educational Measurement
Guillermo Solano-Flores
WestEd/National Center for Improving Science Education
Paper presented at the Annual Meeting of the American Educational Research Association. New Orleans, LA, April 1-5, 2002.
The ProblemThe Need for a Socio-Cultural Perspective in Educational Measurement
Guillermo Solano-Flores
WestEd/National Center for Improving Science Education
Paper presented at the Annual Meeting of the American Educational Research Association. New Orleans, LA, April 1-5, 2002.
Psychometricians have long been aware of cultural issues in assessment. However, their methods and theories have failed to incorporate cultural anthropology into their reasonings and criteria for test validity. Researchers concerned with social and equity issues in assessment have long questioned the fairness of current assessment practices. However, their language and reasonings rarely include methods and reasonings from psychometrics. As a result, these two groups of professionals speak to different audiences and use and recognize different methods and paradigms.
Perspective
The use of cognitive theory-based procedures in test validation has opened the door for multidisciplinary approaches in the field of assessment. However, research on cognitive test validity has overlooked the fact that culture and society shape formal, school learning experiences. To consider student cognition properly, test development and validation procedures should incorporate the Vigotskyan, socio-cultural perspective of cognition as a social construction.
Conceptual Framework
The concept of cultural validity was postulated as a new form of test validity in assessment (Solano-Flores & Nelson-Barber, 2001). Cultural validity is the effectiveness with which an assessment addresses the socio-cultural influences that shape student thinking and the ways in which students make sense of items and respond to them. These socio-cultural influences include the sets of values, beliefs, experiences, communication patterns, teaching and learning styles, and epistemologies inherent to the students' cultural backgrounds, and the socioeconomic conditions prevailing in their cultural groups.
To attain cultural validity, the process of assessment development must consider how the socio-cultural context in which students live influences the ways in which they make sense of items and the ways in which they solve them.
The Research Project
"Assessing the Cultural Validity of Science and Mathematics Assessments," a project funded by the National Science Foundation, addresses a fact that cultural anthropologists have acknowledged for years: culture and society have a critical role in cognitive development and shape thinking processes. If culture and society shape an individual's worldviews, then it is reasonable to expect that culture and society may influence the way in which students interpret science and mathematics items and respond to them.
Participants and Instruments
1096 foruth- and fifth-grade students from 12 localities in the United States participated in the study. They constitute a good sample of the rich ethnic mosaic of this country:
| Social Context n | ||||||
| Locale | Ethnicity | SES | Location | P&P | INT | |
| R (rural) | Micronesian | M | Micronesia | 146 | 12 | |
| S (suburban) | Caucasian | H | Southern CA | 80 | 15 | |
| U (urban) | Black | M | DC | 93 | 18 | |
| R | Latino | L | Central WA | 171 | 20 | |
| R | Yup'ik | L | Southwestern AK | 123 | 33 | |
| R | Asian | M | CA Bay Area | 116 | 21 | |
| U | Haitian | L | NY | 119 | 21 | |
| S | Caucasian | H | Northern CA | 112 | 53 | |
| R | Latino | L | Northern CA | 42 | 18 | |
| U | Black | L | CA Bay Area | 54 | 19 | |
| R | Indian | L | New Mexico | 0 | 41 | |
| U | Black | L | Maryland | 40 | 60 | |
| 1096 | 331 | |||||
Students completed a short test with two science and two mathematics items selected from the 1996 NAEP science and mathematics exercises, which have been used in the past in large-scale testing. Students were also given a questionnaire on activities they engage in at school and outside school and that may be related to the content of the items. Sub-samples of students from each cultural group were also interviewed individually. These students were given one of the items and asked to provide concurrent and retrospective verbalizations of their thinking as they solved the problems. Then, they were interviewed individually to determine how they related their everyday personal experiences to the content of the items.
Scoring and Coding Procedures
The items were triple-scored with NAEP scoring rubrics and procedures, and the interviews were transcribed. At this time, the interview coding is still in process. The coding system is based on Rogoff's analytic framework, which defines three planes of social participation, passive learning (observer), helper (learner), and participatory appropriation (actor). This analytic framework enables the research team to identify patterns of socio-cultural activity in relation to both the inferred cognitive activity and the content of the items. The analytic framework allows the research team to examine how observed different patterns of socio-cultural activity may account for performance (p-values) statistically significant differences across cultural groups.
Results
High inter-rater reliabilities (r >.85) were obtained for the scoring of student responses to the NAEP items. Although interview coding is still in process, available data from coded interview segments indicate high percentages of agreement between coders (>85%).
Preliminary results indicate that student personal experience and epistemology, rather than formal instruction, is a major factor that influences how students interpret items and respond to them. The results indicate that informal, first-hand experiences at home and within their communities shape the way in which students make sense of science and mathematics items. Items that have a higher contextual connection to personal experiences can affect how students interpret items and respond to them.
Example 1
One of the NAEP items used in the project is Lunch Money:
| Sam can purchase his lunch at school. Each day he wants to have juice that costs 50¢, a sandwich that costs 90¢, and fruit that costs 35¢. His mother has only $1.00 bills. What is the least number of $1.00 bills that his mother should give him so he will have enough money to buy lunch for 5 days? |
One of the questions asked to students when they are interviewed is What is this item about?
Student responses to this question were coded according to whether they focused on: (1) textual information provided by the item, (2) contextual information provided by the item, or (3) perceived concepts addressed by the item.
The table shows the percentages of students from three sites whose responses were coded in each category:
| Category | Example of Student Response | CNMI | Central WA | AK |
| Text /Basic Skills | [The item] is asking me what is the least number of $1.00 bills that Sam's mother should give him so he will have enough money to buy lunch for 5 days. | 67 | 33 | 30 |
| Context/ Personal Experience | [The item] is about a boy who needs to buy his lunch. | 17 | 33 | 20 |
| Content/ Concepts | [The item] is about adding and multiplying to figure out the total amount of money and the number of one-dollar bills. | 17 | 33 | 50 |
The table illustrates how students from different cultural groups show different patterns in the way they interpret the content of the item. Similar results have been obtained for other items.
Example 2
One of the NAEP items used in this project is Metals:
| Many things are made of metal, such as pots, pans, tools, and wire. Give two reasons why metals are used to make many different things. |
Many students fail to provide a correct response. However, their explanations during the interview reflect a good knowledge of metal properties. Some students interpret "reasons" as "examples." To others, "why metals are used to make many different things" makes no sense unless the question reads, ...Give two reasons why metals, not other materials, are used to make many different things.
Implications
Current test development practices are assumed to ensure that items are written in a manner that all students interpret them in the same way. However, our results indicate that students from different cultural groups do not necessarily interpret items used in standardized testing in the same way.
When the coding for all cultural groups is completed, it will be possible to compare the interpretations of mainstream and non-mainstream students. This will be helpful in determining whether the wording of items reflects the thinking of the mainstream culture and may be biased against cultural minorities.
Discussion
The notion of cultural validity provides a perspective that considers the reasonings and empirical evidence from three areas, cognitive psychology, cultural anthropology, and psychometrics. Given the possibilities of the concept of cultural validity and the empirical evidence collected, our results can contribute to the field of assessment in two ways. First, they show that we must enrich our thinking and practices in test validation in a manner that they promote more equitable testing in science and mathematics. Second, they show that attaining cultural validity may require a paradigm shift in assessment-the adoption of new assessment practices and reasonings.
The concept of cultural validity has serious implications for both classroom-based and large-scale assessment as a reasoning tool that contributes to obtaining more accurate information about the achievement of students from certain cultural minorities. The approach used in this project honors the importance of systematically collecting information on the ways in which culture influences the thinking elicited by science and mathematics items. If culture influences the way in which students interpret and solve test problems, then not considering cultural differences may penalize students from some cultural groups.
Results obtained so far support the notion that cultural validity is a form of test validity that should be considered systematically in assessment development and testing practices. The results obtained so far also support the notion that measurement specialists should incorporate a socio-cultural dimension into their reasoning and methods.
Related References
Solano-Flores, G. (2001). Worldviews and testviews: the relevance of cultural validity. Paper presented at the European Association of Research in Learning and Instruction. Fribourg, Switzerland, August 28-September 1.
Solano-Flores, G. & Nelson-Barber, S. (2001). On the Cultural Validity of Science Assessments. Journal of Research in Science Teaching, 38(5), 553-573.
Note
This paper reports some of the results from a project funded by the National Science Foundation, grant number REC-9909729. We are grateful to Rebeca Díaz, Jo Ann Izu, and Rachel Lagunoff for their participation in the data collection stage of the project, and to Sharon Nelson-Barber for her collegial support.
Author contact information
E-Mail: wsolano@wested.org
Web: http://www.wested.org/cva/