The NYT has the scoop on the spate of recent testing blunders, and wonders if the rising demand for tests should be met with a rising demand for accuracy and accountability from test developers:
Testing is the buzzword of education these days, with state legislatures and the federal government demanding more of it than ever before. Everything from high school graduation to eligibility for transfers, tutoring and federal aid is tied to the results. But educators and some testing industry experts are warning that the new demands are pushing the limits of the testing industry's ability to provide fair and accurate tests.
When President Bush signed the No Child Left Behind Act in January 2002, calling for increased annual testing in grades three through eight by the 2005-06 school year, the testing industry — dominated by a handful of companies — had just weathered the three most error-plagued years in its history. Researchers at Boston College recently found that last year was hardly better, with at least 18 problems reported, almost matching the total reported between 1976 and 1996.
This surge in testing errors is no joke, but it's also no surprise to those of us who have watched the industry expand at a much faster rate than psychometricans can be trained and standards can be perfected. Testing companies are notoriously close-mouthed about what goes on inside their doors, but part of the problem is that they are expected to provide tests "good, fast, and cheap" - and the problem is that they've had to "pick any two" of those qualities to get the job done. Errors often get caught when test forms are released, but that practice is prohibitively expensive for many states.
Some of the more recent testing criticisms lump big errors in with little ones, as I noted a while back. But even testing defenders concede that the haste in which they are asked to produce good material is the main cause of errors:
Several testing company executives said that the Boston College study reflected an "antitesting agenda" and that it did not distinguish between serious errors and trivial ones. But they agreed with the researchers that haste was the most common contributor to errors. Neal Kingston, the chief operating officer at Measured Progress, said his company had occasionally been asked to devise and deliver new statewide tests in three months — an utterly impossible task, he said.
Is industry regulation the answer?
Concern about this rising tide of testing errors is reviving the long-dormant issue of industry regulation. "We regulate our pet food, and we don't regulate the tests which are making major decisions about the lives of our kids," said Monty Neill, executive director of FairTest, an advocacy group in Boston.
Others have called for an independent oversight panel that could monitor for quality in testing. Professor Madaus, the co-author of the Boston College study, said he preferred that approach to letting the federal government regulate the industry because he feared that politics would taint the professionalism of test evaluation.
Even some testing executives see merit in at least compiling a national database to track testing errors. "Researchers have to hunt and peck where they can to find the mistakes and compile them," said Dr. Kingston of Measured Progress. "A lot of mistakes, quite possibly, don't even get caught."
An independent oversight panel, free from all political bias? A lovely thought, but does such a collection of psychometricians and educators exist?
Posted by kswygert at September 2, 2003 10:55 AM