Several Devoted Readers sent me the news of this big "ooops!":
Mistakes in the scoring of an examination that 18 states use in licensing teachers caused more than 4,000 people who should have passed it to fail instead, the Educational Testing Service said yesterday. The errors may have prevented many from getting full-time jobs as teachers in the last year.
Robert A. Schaeffer, public education director of the National Center for Fair and Open Testing, which looks skeptically on standardized testing, said the grading errors were only the latest instance of quality-control problems in the industry at a time when testing was growing sharply.
Wow, they managed to wait until the second paragraph before bringing in the critics to claim that this is a harbinger of doom, rather than an isolated error. Given that it's the NYT, that's restraint. Also, notice how they also don't mention here that any field that is "growing sharply" is almost always going to produce more errors than one that is stagnant.
Of course, it was the Washington Times that reported Scheaffer's ridiculous quote about how there's no guarantee that anyone in my field is "highly qualified." Funny, but all the psychometricians and test developers I know have Ph.D.s. The demand for testing does mean that we need more qualified people, but it's absurd to insist that any testing error is evidence in and of itself that the people involved were not qualified (all humans, even qualified ones, can make mistakes).
The dog food industry analogy? Rude, bogus, and a cheap shot. Name me one industry that involves the ingestion of any substance by any critter that is not more highly regulated than any psychological testing or assessment industry. For those testing critics too biased to get the picture, let me explain - tests don't kill people. And that's why we don't have something like the FDA overseeing us.
The errors occurred from January 2003 to April 2004. During that time, the test - the Praxis Principles of Learning and Teaching for Grades 7 to 12, called the Praxis P.L.T. 7-12 - was given eight times, to a total of about 40,000 people.
The testing service began notifying state education departments last Friday afternoon that many of those scored as failing had in fact passed, and started calling the candidates themselves on Saturday.
It said it would reimburse the candidates the $115 it cost each to take the test and would also pay them for materials they used to prepare. The cost of test reimbursement alone will be close to half a million dollars.
Tom Ewing, a spokesman for the Educational Testing Service, said that it had noticed lower scores than usual on two administrations of the test, but that "we thought there were valid explanations for why the scores were lower."
"But when we investigated further," Mr. Ewing said, "we discovered that the short-essay questions were being graded more stringently than normal"...
Besides calling state officials and test takers, the testing service has a toll-free phone line (800-205-2626) for more information. A recording at that number yesterday said that the company was "very sorry that this has occurred" and that it was "committed to addressing any concerns this issue may raise for you."
ETS blundered. ETS found the problem. ETS admitted the error. ETS is trying to rectify the mistake, in both the financial and career-impact domains. In my mind, these are signs of an industry that is functioning in a normal, healthy fashion. Glad to see the article close with a quote that is complimentary to ETS. And note, too, that the error was in the "performance assessment" portion of the exam; testing critics often call for such performance-based items due to an irrational hatred of the more reliable and cheaper multiple-choice items.
I see enough complaints about Praxis on the web, though, that I expect this to bring out many, many responses of how unfair the test is, and how this error must prove...something.
Posted by kswygert at July 13, 2004 03:22 PM