Devoted Reader Nick sends along an article about the protest against teacher certification exams in New York City. Not surprisingly, the protestors are claiming that the test is biased against those who failed; also not surprisingly, the NYT reporter who covered this story fails, along with the protestors, to grasp the distinction between bias and impact.
The protesters urged the city to reconsider its firing of 10,000 uncertified teachers over the last five years, saying the test was culturally biased against blacks and Hispanics.
Marc Pessin, a co-chairman of a teachers' group called the Progressive Action Caucus, which organized the demonstration, said the passing rate among blacks and Hispanics was about 40 percentage points lower than that among whites.
If the passing rate is different for blacks, whites, and Hispanics, then this means the distribution of scores are different. The black and Hispanic distributions are most likely shifted lower on the score scale continuum, so that a smaller percentage of those members fall above the passing cutscore than in the white distribution.
This is NOT evidence of bias; indeed, bias can exist where group mean differences do not. No reporter seems to understand this, and no testing opponents want to believe it. The differential passing rate means only that the groups differ on the ability scale. There are many explanations for why that would be; wholesale "cultural bias" is one of the least likely, and, as I'll explain in a moment, one of the most distasteful explanations.
There is not any firm evidence of test bias here, but because black and Hispanic teachers are passing at a lower rate, using the test for certification purposes does have a greater negative impact on those test takers. Impact is, in itself, a neutral term, and not a criticism of a test. Here's why.
Example: Group A and Group B take the driver's licensing exam. 90% of Group A passes; 15% of Group B passes. Is the test biased against Group B? No, but the test negatively impacts Group B. Suppose you then discover that Group A has had driving lessons, while the members of Group B have never sat behind the wheel of a car. Is it a bad thing, then, that the exam has a differential impact on the two groups? Not at all; in fact, the results are evidence of the construct validity of the exam, because the results show that the exam prevents people who don't know how to drive from obtaining licenses.
The analogy holds here. Black and Hispanic teachers do not pass the test at the same rate. It is not racist to say that, for some reason, these teachers are less likely to have the skills necessary to pass the exam. Perhaps they were not taught them; perhaps they did not believe they would ever be tested on them.
It is racist to claim, as the representative of the "Progressive Action Caucus" quoted above does, that these teachers are "culturally" unable to pass this exam based solely on their skin color. Saying that black and Hispanic teachers are not capable of answering multiple choice items for reasons that have nothing to do with their skills, and everything to do with their race, is hardly a "progressive" idea. In fact, the Klan would likely agree with the "Progressive Action Caucus" on that point.
(More on bias and impact here, here, and here.)
What does the test look like? See for yourself. Here's the test framework for the Liberal Arts and Sciences section, and sample test questions can be found here. The questions ask about topics such as equilibrium, linear relationships, a basic understanding of experimental design, and understanding mathematical concepts that have been graphed. The English prep guide is here; sample multiple-choice items begin on page 24. The reading load is substantial, but only people who think teachers shouldn't know how to read English very well would think that's a problematic aspect of the test:
Jose Aguasvivas, a former math and Spanish teacher in the bilingual program at Roosevelt High School in the Bronx, said he was typical of teachers who had failed the test because English was not their first language. "I even have a master's degree in bilingual elementary education," he said. "But the test is very confusing. If the test is in Spanish, then I pass it no problem."
Are you teaching in a Spanish-speaking country? Are you supposed to be instructing students only in Spanish, and never in English? If not, then this justification isn't satisfactory.
Another teacher laments the fact that she failed the test after teaching for 19 years and has had to go on welfare as a result. One could conclude the test was unfair; one could also conclude that this teacher was not very skilled to begin with, as evidenced by her failure to pass the exam, and her failure to find another job.
So, how difficult is this exam? This letter claims that most teachers pass on the first attempt. This page gives the passing scores, but not the total scores on each section. Teachers can retake an unlimited number of times, but must pass each section on one attempt to receive certification (they cannot combine passing sections from multiple attempts). This FAQ from a test prep site claims that teachers need answer correctly only 50-55% of the multiple-choice items in order to pass each section of the exam (they must pass the essay sections as well).
I dispute the claim that the test is biased, but do we know for sure that the exam is useful for identifying good teachers? Not necessarily. At least one study failed to find a link between teacher certification and student test scores - but DID find that the strongest effect on student performance was due to "teacher verbal and cognitive ability," which is certainly something that can be incorporated into a certification exam.
How do you think the "Progressive Action Caucus" would react if NYC suggested adding an IQ test to the certification procedure, or incorporating SAT scores? The fireworks would be fun to watch.
Posted by kswygert at August 28, 2003 10:50 AM