Here's an article in My San Antonio online that does a great job of describing how tests get produced and scored at the Harcourt Assessment organization:
In the second [HA] building behind the main lobby, certain corridors are off limits, and visitors are flanked by helpful, cautious guides. In there is the massive warehouse where millions of pieces of paper are tracked and sorted by radio frequency, with two sets of loading dock doors to segregate incoming from outgoing.
Next to it is the room where multiple-choice answer sheets full of pencil-darkened bubbles are run through grading machines. The humidity in that room is carefully controlled so that the answer pages won't expand or shrink from the exact proportions the computers have been programmed to read.
Upstairs is a great gray room filled with rows of long tables lined with computer monitors. During the winter months, only a handful of people sit there. But as the days grow warmer, students across the nation will apply their No. 2 pencils to millions of late-spring standardized tests. That's when Harcourt fills the long tables with hundreds of part-timers and temporary employees hired to read thousands of answers to essay questions...
Despite the extra thought that's supposed to be tapped by essay questions, the job of grading the answers is tedious and often boring, Pete Loxsom, a former Harcourt employee, said..It didn't help that Loxsom has a low opinion of standardized tests and identified with the more sour notes some test-takers struck.
"In some of my essay questions, I would see 'If you're reading this stuff, it must suck to be you.'" Loxsom said. "I thought, 'You kind of nailed it there, kid.'"
Despite the anti-testing backlash, Harcourt is growing rapidly:
The San Antonio-based company, now a subsidiary of Dutch company Reed Elsevier after almost a century of name changes, mergers and acquisitions, basically doubles its staff of almost 1,300 during the peak scoring season.
The company also added 367 employees last year and plans to keep growing if it can find enough statisticians and psychometricians, who are specialists in the measurement of academic achievement. It wouldn't release figures on what it pays or what it earns but called 2003 "a successful year."
That "if" above is crucial. The number of tests required each year keeps growing, but that's not being met by an increasing number of psychometricians. Even the executive director of FairTest, who's never met a test he liked, is quoted as saying that one of the main causes of scoring errors is the understaffing of psychometricians at companies that are pressured to turn scores around quickly.
Harcourt has apparently started a fellowship program to "woo" more students into the field of psychometrics. ETS, LSAC, and ACT already have similar programs in place (full disclosure: my last two years of graduate school work were supported by an ETS fellowship). Here's a press release advertising the fellowship.
Note the names of the four doctoral fellows who spent the summer at Harcourt in 2003, learning the ropes. America is a mecca for would-be psychometricians from around the world (the Netherlands is a close second).
Posted by kswygert at January 20, 2004 12:51 PM