Tech Central Station has another good education-related article up; this one, by Howard Fienberg, is about California's failed Academic Performance Index, or API. Fienberg's description of the problem:
The regime under which improving schools and their teachers receive cash awards fails to include the test scores of one out of every 5 students in the state. According to an investigative series from the Orange County Register, that omission results in an average 20 point margin of error in schools' test scores. It may be that some California schools which got awards did not actually earn them, while other schools were unfairly left wanting...
Schools get an API score out of a possible 1,000 points, based on the Stanford 9 standardized test. Score improvements are rewarded with cash. But it seems that some details were never really worked out - the margin of error was never discussed, except for in passing at a September meeting that year...
Not only are some students not taking the exams, but many others that do take them have their scores removed afterwards. California claims that 98-99 percent of all its students are tested, but it appears that, on average, only 82 percent of students are counted in the end
Both of these problems are serious ones. For starters, the API score is high-stakes, because the cash awards are based on it. But every measurement contains some error, and you cannot make even low-stakes decisions based on measurements without some knowledge of the amount of error in your data. Does California know why the margin of error is 20 points? That's not huge, considering the 0-1000 point range, although if in reality few schools score below 500, or 600, the 20 points becomes more problematic. Assessment of error, and allowance for it when making a high-stakes decision, should have been two of California's first priorities.
I've also blogged before about California's tendency to leave out certain students' test scores, and Fienberg doesn't find it any more acceptable than I do. He also comments on the "grouping" mechanism that was certainly intended to benefit minorities but has ended up having the opposite effect:
The API awards system was not only designed to reward improvements in overall scores, but also scores in each major ethnic and racial group, as well as among the poorest students. The OC Register found the unintended consequence of the 'grouping' system was favoritism for the least diverse schools when it came to getting awards. The paper compares it to the difficulty of winning consecutive coin tosses. "The fewer groups, the fewer coin tosses a school has to win." About 58 percent of schools statewide with only one major ethnic/racial group won awards in 2001, compared to almost 29 percent of schools with four or more groups. As a result, mostly white schools received an average of $21 per student, while the most diverse schools nabbed an average of only $9 per student...Such disparities in outcome would not be quite so disturbing were it not for the API's margin of error. The complexity of the state's 'grouping' system creates extra problems on top of the regular error. Error rates grow as the API measures smaller and smaller sub-groups, inevitably obscuring real gains and losses. While schools with less than 100 students don't receive API awards because of the unreliability of their scores, the OC Register points out that the scores of groups can include as few as 30 students.
My favorite attributed comment is at the end of the article:
State officials told the OC Register they didn't disclose the API's error rate for three years because it would have been too confusing to the public.
Wow. So, we' re supposed to believe that California has ignored the margin of error and bollixed up their API results for three years in order to avoid confusing the public? Yeah, right. How about, "to avoid enraging the public", because margin of error is not a difficult concept to understand, or to explain, and "the public" might be pretty pissed to discover just how badly California is mismanaging the state education budget.