NC scores on hold?
Looks like there's a problem with North Carolina's fourth- and seventh-grade reading tests - the scores are on hold for two years. The reason? "An unexplained plunge in fourth-grade scores and a smaller slump for seventh-graders convinced officials there was a problem with the test."
The writing prompt may have been inappropriate for the fourth-graders. A multiple-choice test may be added, so that the score doesn't rest solely on the essay. However, I think the largest problem is with the field-testing guidelines. Field-testing items is one of the long-standing bugaboos in standardized testing. Items that have just been developed must be exposed to test-takers in order to gather information, such as the item difficulty and discrimination, and what the item adds to the reliability of the test as a whole. Item pre-testing is also crucial to catch differential item functioning (which is when subgroups with similar performances answer an item very differently), inappropriate distractors, ambiguous wording, and so on. If the item is flawed, no one is disadvantaged, because field-testing of items doesn't count. But it's crucial in good item pre-testing that students take the items very seriously even though they don't count towards a score, and that's the Catch-22.
It looks like the field-testing of the NC writing items was seriously flawed. The students didn't take the items seriously and resented the extra time taken out of their schedules. I was surprised by this comment - "Only about one-quarter of students passed [the items during field-testing] -- but field-test scores are traditionally so low that it didn't raise an alarm". Field-test scores usually are lower, true, but they shouldn't be so low that you can't get an accurate idea of how the item will perform when it's given for credit. Here, the low field-test scores may have been a warning that the prompts were too difficult or inappropriate.
Thanks to Concerned Reader Jessica S. for sending me the link; Fark.com caught the story as well.