The pressure of tests
Dana Tofig of the Atlanta Journal-Constitution wonders how many testing errors are too many, and whether the testing companies need more regulations in place:
Some high school seniors in Minnesota couldn't graduate with their classmates. More than 8,000 students in New York City were improperly sent to summer school. Closer to home, nearly 600,000 Georgia students won't take the state's most important standardized test this year...Over the past few years, the results of several other standardized tests across the nation also have been tainted by mistakes made by private testing companies.
The errors in the Georgia test that I mentioned last week are rehashed:
Most recently, on March 27, state Superintendent of Schools Kathy Cox announced that some questions used on practice exams for the Criterion-Referenced Competency Test, the state's curriculum test, had appeared on the actual tests that were to be given to students this month. The tests were scotched in all grades except four, six and eight, and the state is still sorting out how the error occurred.
Georgia education officials say one of two testing companies -- Measured Progress or Riverside Publishing -- is responsible. Officials from Measured Progress said they are helping the state investigate and expect some type of answer next week. Officials from Riverside Publishing, a subsidiary of publishing giant Houghton Mifflin, would not comment.
I don't blame Georgia school officials and parents for being upset. These are big problems. However, as the article makes clear, there are millions of children tested each year without error, and the errors that have occurred might simply be the unavoidable byproducts of the vast increase in the amount of K-12 testing in the US:
The issue might be as simple as supply and demand. With "accountability" the buzzword in education, the demand for testing has skyrocketed during the last several years..."Because of accountability and the movement across the country with No Child Left Behind, we have relied on two or three very large companies for testing needs," said Ellen Cohan, associate superintendent for teaching and learning with the Forsyth County school system. "It must be overwhelming for them"...Stuart Kahl, president of Measured Progress, said the testing industry has started to expand to meet the demand...For instance, Kahl pointed to the Educational Testing Service. The Princeton, N.J.-based company once specialized in higher education entrance exams such as the SAT but has now entered the k-12 testing business.
Kahl said the biggest challenge facing the industry is time. He said the companies need to create individualized tests based on each state's curriculum. The companies used to have years to develop tests, he said, but that's changed. "Instead of taking three or four years to develop an assessment system, you've got three or four months," Kahl said.
When the deadline is that short, it's very difficult to make sure that the usual item review, field-testing, and QA processes are in place. What's more, the article doesn't even touch upon one of the most extreme imbalances of supply and demand in the field of testing - a severe shortage of psychometricians. To my knowledge, there have been no articles published about this, but I've been aware of the situation for years. I've yet to work at a place that wasn't hiring - and by hiring I mean "desperately looking for qualified psychometricians and research assistants" - and I know of no unemployed or under-employed psychometricians. I also know very few, especially ones in my age group, who haven't changed jobs in the last five years, because there are so many positions open, with more being created each year.
One reason for this shortage is that, despite the recent explosion of tests testing, the requirements for being a psychometrician haven't been modified, and large-scale high-stakes testing requires lots of psychometricians (and research assistants who are studying to be psychometricians). A psychometrician must possess a Ph.D. (a Masters isn't enough), and it's almost always in Quantitative Psychology or Educational Measurement (or, something very similar with a different name).
In the year 1999-2000, 44,808 people received doctoral degrees in the U.S. Over 1600 were in English; over 1100 were in Mathematics; almost 7000 were in Education. In Psychology, which is the field most psychometrians receive degrees in, 4,310 students received Psychology Ph.D.;'s, and and according to this APA report, in that same year, there were over 5,700 students enrolled in Psychology Ph.D.-level programs in the US. Would you care to guess how many of those enrolled Psychology Ph.D.'s were in my field?
Thirteen.
Yup, that's sure to keep up with the rising tide of testing. Meanwhile, the American Educational Research Association (AERA) alone has 35 research-related position postings - and that's just the positions that are officially listed by an organization that's willing to do a nationwide search.
Sure, we can look at students from other programs, such as Educational Measurement, but the numbers don't improve much. Here's a survey of those receiving doctorates in 2000; out of the 41,368 students sampled who received Ph.D.'s that year, in addition to the 8 Quantitative Psychology Ph.D.s (shown on page 103), there were 13 Ph.D.'s awarded in Psychometrics, 55 awarded in Educational Research Methods (p.104), and 45 awarded in Educational Assessment. What's more, this document shows trends over the last 10 years, and no increase in these numbers is visible.
You know all these new K-12 testing jobs? Some of them pay very well, and almost all of them pay much better than academic positions. No new Psychometrics professors means no new Psychometrics students.
Bottom line? I'm surprised we haven't seen more errors, and more articles critical of these errors. As the demand for testing increases, companies are operating with sub-optimal staffing, and the trained psychometricians who are out there aren't being given enough time to do their jobs correctly. Would national regulation help? I seriously doubt it. A national regulatory board could help develop national guidelines, but there are already informal "industry-wide" standards that any legitimate company would impose upon itself. National regulation also takes power away from the states. It's not that testing companies are deliberately trying to cut corners and produce error-ridden tests - quite the opposite. Without an increase in the number of professional psychometricians to staff the testing companies, all the regulations in the world won't do much good.