Number 2 Pencil

Back in the saddle

Howdy, folks! Starting on April 6, 2008, I will be putting up a "testing roundup" post every weekend at Joanne Jacob's site. Swing on by and put in your two cents!

The first post is here.

UPDATE: And by the way, I have to point out this article in the NYTimes about psychometricians, which many of you probably saw two years ago. I just love that it came out on - yes - the day before my wedding day. Which is why I missed it at the time.

Blinding Indiana with science

Is this good news or bad news for Indiana? You be the judge:

More than half of Indiana's seventh-graders passed the state's mandatory science exam, administered to that age group en masse for the first time last fall. Fifty-two percent of the state's 80,863 seventh-grade students passed the science assessment, a new section in the annual Indiana Statewide Testing for Educational Progress-Plus exam. That left 47 percent below the benchmark, said Mary Tiede Wilhelmus, spokeswoman for the Indiana Department of Education; 1 percent of the results were unscoreable.

"We certainly know we need to do better, but this is a starting point," said Tiede Wilhelmus. Two areas that gave kids trouble, she said, were sections on the nature of science and technology -- which included scientific investigation -- and one focusing on the physical universe.

Indianapolis schools fared much worse, with only 18% passing the exam. A sample exam is here. It's pretty open-ended. I had to guess on the first one - I remembered the orbits as being circular rather than elliptical, and that is one of the correct answers.

Deep breathing for the PSSA

One suburban Philly teacher leads tai chi classes to help prepare kids for the state standardized exams:

A Centennial teacher is utilizing the ancient martial art of Tai Chi Chuan to help his students get ready for the state's standardized mathematics and reading tests.

“I want you to concentrate. Think about what you are doing. Breathe in and breathe out,” Joseph Pisacano, a fifth-grade teacher at Everett A. McDonald Elementary School in Warminster, said Thursday morning as he and the students made smooth, circular motions with their arms and hands.

The class went through the Tai Chi relaxation exercises as calm, soothing music played quietly in the background.

“Think about how wonderful you are going to do on the test and how easy it's going to be. Think about all that you have learned this year. Take all of that, and use it,” the teacher said as the class wrapped up the exercise session by sitting in a meditative position on the floor.

As long as he's making sure to actually teach the material along with some relaxation exercises, I'm all for it. This approach is worlds better than the chicken-little squawks you see from some educators who are so stressed out about testing that they end up stressing out students as well.

Fix the system where it's broken

The idea of standardized testing in college is still gaining attention - most of it negative:

A parade of college presidents will appear before a federal higher-education commission meeting in Boston tomorrow, and early signs suggest it will be a lively, even contentious scene. Texas businessman Charles Miller, chairman of the commission appointed by Education Secretary Margaret Spellings, has made waves by suggesting that some kind of standardized testing would help measure whether college students are taught well. There is no formal proposal yet, and Miller has stressed in press interviews that there would be no single test for every school. Still, the idea has alarmed many educators.

Susan Hockfield of MIT, who had lunch with the Globe editorial board last week, didn't mince words when asked about the testing notion. ''I think it's a terrible idea," said Hockfield, who is scheduled to testify tomorrow. ''Higher education needs help, but what is really broken is K-12 education. We need more high school graduates who can understand and do math."

In other words, there's a problem, but it can't be fixed in college. I agree with Dr. Hockfield, but I wonder if she knows how strong the opposition can be to "fixing" anything in the K-12 system:

At School Without Walls and two other high schools where I am a guest teacher -- Wilson High School in the District and Bethesda-Chevy Chase High School in lower Montgomery County -- I have never given a test. I respect my students too much to demean them with exercises in fake knowledge.

Tests represent fear-based learning, the opposite of learning based on desire. Frightened and fretting with pre-test jitters, students stuff their minds with information they disgorge on exam sheets and sweat out the results. I know of no meaningful evidence that acing tests has anything to do with students' character development or whether their natural instincts for idealism or altruism are nurtured.

I have large amounts of evidence that tests promote the opposite: character defects.

Good luck convincing this teacher that reading and math basic skills can and should be assessed. She's much more concerned about "idealism" and "altruism." How far do those get one at MIT, I wonder? As Betsy's Page notes, this teacher teaches a class on peace. Most of his students love him, it seems - and wouldn't you, if testing was banned? - but not all:

At Bethesda-Chevy Chase, Peace Studies is taught by Colman McCarthy, a former Washington Post reporter and founder and president of the Center for Teaching Peace. Though the course is taught at seven other Montgomery County high schools, some say B-CC's is perhaps the most personal and ideological of the offerings because McCarthy makes no effort to disguise his opposition to war, violence and animal testing.

Saraf and Avishek Panth, also 17, acknowledge that with the exception of one lecture they sat in on this month, most of what they know about the course has come from friends and acquaintances who have taken the class. But, they said, those discussions, coupled with research they have done on McCarthy's background, have convinced them that their school should not continue to offer Peace Studies unless significant changes are made. This is not an ideological debate, they said. Rather, what bothers them the most is that McCarthy offers students only one perspective.

Of course he does. This is a crusade for him. One that doesn't involve anything as nasty and dehumanizing as testing (and that's even funnier than "high comedy," according to one Devoted Reader who forwarded the link). After reading his WaPo diatribe about testing, it's hard to believe that he actually is "welcoming of conservative dissention" on any topic.

Do high-stakes results look the same as low-stakes results?

When the stakes are high, do the results make sense? Jay Greene, Marcus Winters, and Greg Forster address the issue:

Several objections have been raised against using standardized testing for accountability purposes. Most concerns about high-stakes testing revolve around the adverse incentives created by the tests. Some have worried that pressures to produce gains in test scores have led to poor test designs or questionable revisions in test designs that exaggerate student achievement...Others have written that instead of teaching generally useful skills, teachers are teaching skills that are unique only to a particular test...Still others have directly questioned the integrity of those administering and scoring the high-stakes tests, suggesting that cheating has produced much of the claimed rise in student achievement on such exams..

Most of these criticisms fail to withstand scrutiny...This study differs from other analyses in that it focuses on the comparison of school-level results on high-stakes tests and commercially designed low-stakes tests. By focusing on school-level results we are comparing test results from the same or similar students, reducing the danger that population differences may hinder the comparison. Examining school-level results also allows for a more precise correlation of the different kinds of test results than is possible by looking only at state-level results, which provide fewer observations for analysis...

The conclusion? That, within a school, correlations between high- and low-stakes tests tend to be large and postive:

The finding that high- and low-stakes tests produce very similar score level results tells us that the stakes of the tests do not distort information about the general level at which students are performing. If high-stakes testing is only being used to assure that students can perform at certain academic levels, then the results of those high-stakes tests appear to be reliable policy tools. The generally strong correlations between score levels on high- and low-stakes tests in all the school systems we examined suggest that teaching to the test, cheating, or other manipulations are not causing high-stakes tests to produce results that look very different from tests where there are no incentives for distortion.

MEAs for little Mainers

Maine is now testing its third-graders regularly, and teachers are having to walk a fine line of test prep and emotional support:

They want the students to take the tests seriously, because they're used by the state and federal government to measure whether schools are teaching students math and reading effectively. But they don't want to pressure young students new to the game of high-stakes testing.

"We tell them to just do their best," said Hall-Dale Elementary School third-grade teacher Maureen Mathews. "You want to them to know it's important. But you don't want to make them nervous."

Love the photo of one neophyte examinee, who appears to be a direct descendant of Dame Judi Dench.

Temp-orary insanity on the FCAT

Critics are howling about ten-buck-an-hour temps grading FCAT essays:

Critics were fuming Friday after learning the FCAT -- the standardized test that will leave a permanent mark on the academic future of thousands of Florida students -- will be graded by $10-an-hour temporary workers who are required only to have a week's training and a bachelor's degree.

"It's just incredible to me that after all of the pressure that is placed on me to maintain my teaching credentials, the countless hours spent in workshops, and then they turn around and hand these tests off to be scored by a bunch of temps," said David Worrell, president of the Leon Classroom Teachers Association. "It's just insulting."

The DOE says that many of the workers have teaching experience and are very familiar with the exams. Keeping permanent full-time graders would certainly up the costs of the exam. Others say that criticizing the temps misses the point:

Of the many legitimate concerns raised about the FCAT over the years, this is the least of them. Temps are used to grade other important exams, such as the ACT college entrance exam, and the FCAT graders will have bachelor's degrees and training. Those grading essays will handle questions that relate to their college degree. Essays are to be graded twice, and possibly a third time.

The real problem is that, under Gov. Jeb Bush, the FCAT has been wielded like a cudgel in the evaluation of school quality - and now, teacher quality.

In other words, they're fine with how the essays are graded; they're just not happy with the FCAT use as a whole.

A cheapened college will still cost a bundle

Hey, look, it's a foolproof method for making college educations cheaper!

Oh, wait, that's not what they meant. Never mind. I'll just point out that the author is against college-level standardized tests, in part because, "achievements in math and science will speak for themselves." Couldn't that theory be used as an argument to test everyone who doesn't take those kind of courses in college?

FairTest feels the crunch

Well, well, well.

For more than 20 years, FairTest, a small nonprofit group headquartered on the second floor of an old house here, has been the No. 1 critic of America's big testing companies and their standardized tests. In 1987, when FairTest began publishing its list of colleges that did not require applicants to submit SAT's, there were 51; today there are 730, including Holy Cross, Bowdoin, Bates, Mount Holyoke and Muhlenberg.

...for all FairTest's impact, its days may be numbered. Never before has standardized testing so dominated American public education, thanks to the 2002 federal No Child Left Behind Law. Every child from grade 3 to high school must now take state tests. And the Bush administration is considering extending those tests to colleges.

"With N.C.L.B., a lot of people feel the debate is over," said Monty Neill, director of FairTest, officially the National Center for Fair and Open Testing. "The attitude seems to be, 'Testing is so pervasive, what's the point?' " Support from foundations has virtually dried up and individual donations have not made up the difference. "Our board has seriously discussed whether to fold the operation," Mr. Neill said.

I find this a pretty revealing comment. There's always a need for testing to be scrutinized, for tests to be evaluated, and for the public to be informed. But I've always sensed that FairTest's commentary was always anti-any-testing, not pro-good-testing. Now that testing is so pervasive, it's not helpful to bash tests rather than inform the public. ETS's president seems to agree:

Kurt Landgraf, the president of the testing service, which administers the SAT, wrote in an e-mail message: "Perhaps if they had been more attuned to the public's support for using tests to help teachers teach and students learn, then they might have had wider support."

Further along in the article, I don't quite get the point of NYT reporter Michael Winerip listing this as though it's a bombshell:

In a recent newsletter, FairTest printed an analysis of SAT results, using, and crediting, College Board research showing the direct correlation between family income and SAT scores. For every extra $10,000 a family earns, children's combined math and verbal scores go up 12 to 31 points. So children whose parents earn $50,000 score better on average (a combined 996 SAT) than students from families who earn $40,000 (967) but worse than students from families who earn $60,000 (1014).

Okay, who alive today doesn't know that kids with more money tend to have more educational advantages? It makes sense to me that kids from wealthier homes do better on all educational indices; if they didn't do better on the SAT, parents would question the efficacy of private schools and tutoring. Why this is being mentioned here as though it's surprising knowledge - or a valid test criticism - is beyond me.

At the same time, correlation doesn't equal causation. Just because A and B are correlated, that doesn't mean A causes B. B might cause A, or some C could be causing both to happen. Smarter parents might make more money, and their kids get both the nature and nurture benefits. We are trying to close the gap by offering all students better opportunities, but a test that doesn't reflect when kids know more material, either by virtue of schooling or parental largesse, is a pretty useless test.

How about "teaching with," instead of "teaching to"?

Jay Mathews offers a spirited defense of a process much maligned in the education world - "teaching to the test":

When we say "teaching to the test," we should acknowledge that we are usually not talking about those drill fests. Rather, we often use the phrase to refer to any course that prepares students for one of the annual state assessment exams required under the No Child Left Behind Act. For reasons that escape me, we never say a teacher is "teaching to the test" if she's using a test she wrote herself. We share the teacher's view that what she is doing is helping her students learn the material, not ace the test. But if she is preparing the class for an exam written by some outsider, the thinking goes, then she must be forced to adhere to someone else's views on teaching and thus is likely to present the material too quickly, too thinly, too prescriptively, too joylessly -- add your own favorite unattractive adverb...

...Conversations about this would go more smoothly if we didn't have such distorted views of what teaching to the test means. We might instead turn the discussion to what methods of instruction work best or how much time our children should spend studying.

Higher stakes, higher pay

The more-pay-for-higher-scores plan in Florida has passed the Board of Education approval stage:

The Florida Board of Education unanimously approved a plan Tuesday that will give some teachers bonuses based solely on their students' performance on standardized tests.

As early as next year, the plan will award the top 10 percent of teachers in each school district a 5 percent bonus based on learning gains shown on the Florida Comprehensive Assessment Test.

If districts want to reward more teachers, they can. But there may not be state funding for it, officials warn. The plan also will require the state to create exams or other assessments in every subject not covered by the FCAT.

Whew. Possibly no funding and definitely more testing? Gee, wonder if this plan is causing any controversy. (Obviously, I'm being sarcastic here.)

Higher FCATs, more bucks?

Florida Education Commissioner John Winn wants to get serious - about tying teacher bonuses to FCAT scores:

In Miami-Dade and Broward counties, teachers could earn bonuses ranging from $1,710 to $4,150 per year on salaries that range from $34,200 to $83,070.

The policy would add another set of consequences to Florida's high-stakes accountability system, which already determines school grades, high-school graduation and whether students can progress from third to fourth grade.
Winn said the Effective Compensation plan, which he dubbed E-Comp, would encourage teacher improvement, reward excellence and bolster recruitment and retention.

The critics, they disagree:

E-Comp would replace existing bonus systems in Miami-Dade and Broward, both of which Winn said were unacceptable.

In Miami-Dade, a 5 percent bonus is given to all teachers at 28 schools that have the largest gains in FCAT reading and math scores. UTD President Karen Aronowitz said that system is fairest because many teachers contribute to a student's success. ''When we send firefighters to a fire, do we pay them differently based on who handles the hose?'' she asked.

No, but we do tend to get rid of those who don't pick up their end of the hose, especially if the end result is a house burnt down to the ground. If the public were secure in the knowledge that bad teachers would not only not share in school-wide compensation, but would get dismissed to boot, I'm not sure that merit pay for especially-good teachers would even be an issue.

New GRE set to go live this October

The new GRE grows ever closer:

According to the ETS Web site, the changes will better gauge students' preparation for graduate school by measuring general academic skills with more precision than in the past. A single 30-minute verbal section will be changed to two 40-minute sections. Sections on analogies and antonyms will be removed, while new sentence equivalence questions will be introduced and critical reading sections will be expanded. Quantitative reasoning -- lengthened from one 45-minute section to two 40-minute sections -- will include less geometry and more data interpretation and word problems. The test will be graded on a scale of 120-179, as opposed to the current 200-800 scale.

Some students resent the newer, longer length:

...Sam Penziner '07, who is also planning to attend graduate school, said he thinks the longer exam will measure test-taking stamina rather than skill. "Making the test longer emphasizes factors like endurance and stress that affect performance," Penziner said.

And graduate school doesn't require endurance and good stress-coping strategies?

Judging a district by its scores

University of Arkansas at Fayetteville’s education researchers doubt that standardized test scores are the best indicators of school district performance:

The statistical analysis — which will be presented today to the Arkansas Association of Educational Administrators — concluded that Arkansas students are achieving slightly better than the nation as a whole. And several Arkansas districts not typically recognized for their academic excellence top the scale of high performers...

“The School Performance Index in Arkansas” takes into account student demographics and the levels of affluence and education in a community. It predicts what student achievement in a school or district should be on the basis of those factors and then compares those projected achievement levels to actual standardized test results.

So, if I understand this, they're predicting how well a school should do based on various demographic and SES levels, and then comparing those predictions to the real test scores. They're concluding that raw test scores shouldn't be used to compare schools, but instead should be adjusted to show how well the school is doing given all these predictor variables, so that schools with students who are predicted to do poorly should not be considered bad schools if they produce mediocre test scores. I'm not sure I agree with that conclusion.

One interesting side finding:

The school analysis, which Greene said could be further refined by the state, showed that school performance on the Iowa Test is “partially” affected by the level of household income, educational attainment of adults, and the percentage of married families in a district. The scores are “substanially affected” by the percentages of black students and students who qualify for reduced meal prices at a school.

In contrast, the study concluded that school performance is not affected by the size of a school or a district or by the amount of money spent in a district.

Gee, wonder why the headline for this article wasn't, "Schools don't need more money to perform better?"

Tracking the CAPTive students

A UMass-Amherst research claims the Connecticut Academic Performance Test is extremely predictive of college success - even more so than the SAT:

Stephen Coelen, a researcher from the University of Massachusetts at Amherst, tracked 32,653 members of the Class of 1998, comparing how well they did as sophomores on CAPT to how many applied to, enrolled in and did well in college. On every measure, he found the higher the CAPT score, the more students were likely to go to college, avoid remedial courses in college, get higher grade point averages in college and graduate.

When matched against SATs — the College Board exam students take to predict college success, Coelen said both exams helped explain student success in college. Of the two, Coelen said CAPT "was always correct. SAT was not always correct.

Interesting. The "going to college" part could, I think, be affected by the possibility that those who score high on CAPT in 10th grade spend the next couple of years being groomed by teachers for college. If their CAPT scores affect their high school class placement or treatment in any way, then it wouldn't be surprising that CAPT would correlate with college attendence - it would be one of the predictors of it.

Interesting also to see that the CAPT apparently has a high positive correlation with college grades, but given the outcry we hear these days about grade inflation, one wonders if this is really a positive thing about the exam.

Commissioner of Higher Education Valerie Lewis said the study proves the value of CAPT in predicting college success and should be recognized by college admission staffs as a valuable piece of information when they admit students.

Lewis also found it startling that 10 percent of students who score very high on CAPT never show up in college. That means some talent is going untapped and underdeveloped...

I find 10 percent startlingly low We're not being told what "very high" means, nor do we know the shape of the distribution. The study was composed of around 32,000 kids; if "very high" means the top 5% of scorers, we're talking about less than 200 smart kids from Connecticut in that graduating year who passed on college. I would think that family issues, financial issues, health issues, and lifestyle issues would affect that preclude college would affect at least 10 percent, maybe more. These are people, not automatons, and if they were that smart, they may have well decided that they wanted to do something other than pay thousands of dollars a year for additional education.

The Empire State of Testing

Third- through eighth-graders in New York must now take state standardized exams, and for three of those grades, the tests will determine promotion:

Today, grades three, four and five had a multiple choice test. Tomorrow the same grades will listen to stories, and then write about what they've heard. On Thursday, 4th graders only will be asked to read a passage and then write about it.

This year marks the end of citywide tests and the start of statewide testing for all students in grades three through eight because of the federal law 'No Child Left Behind.' Also this year, state standards are higher.

You can look through the PowerPoint presentation here that gives an overview of the exams. Core curricula in English and Math are also available. A sample exam for Grade 3 looks pretty simple

More testing in Florida

More testing might be on the way in Florida - and that could be a good thing:

Florida high school students may someday have to take end-of-grade tests in history, literature, biology and other key subjects -- possibly in addition to the FCAT. Members of a state task force on high school reform are suggesting the tests as a way to make sure students are really learning what the state says they are supposed to learn...

New York and Texas already use similar tests, and some Florida school districts have adopted them, too. ''We're looking for something that's going to help students achieve at a higher rate, not looking to multiply the number of tests out there,'' said state education Commissioner John Winn. "But an end-of-course test is a way to have some consistency in proficiency level.''

All the psychometrics you need

Chett at ReformK12 just pointed me to a nifty online list of psychometric resources - The Statistics Resources for the Center for Research, Evaluation, and Program Development, Boston Graduate School of Psychoanalysis. I'd recommend the books on IRT, except that I haven't read them - I'm old school and rely on the classic Hambleton & Swaminathan IRT bible. I actually rely on a lot of "bibles" that aren't listed here, like Educational Measurement and Psychometric Theory. A tad ironic that I mainly read these old books, since I'm the book review editor for a measurement journal, but oh well.

I also would be remiss if I did not mention the IRT software packages BILOG and *cough*my-advisor-created-this*cough* MULTILOG. Even I can't claim they're easy to use - unless you're fluent in FORTRAN, and a frightening percentage of psychometricians are - but I see them get a lot of use in testing organizations.

Putting music education to the test

The tweetles, trills, blats, waa-waas, and crashes of Florida's band students may soon be measurable via a standardized exam:

It won't resonate as loudly as the FCAT, but Florida schoolchildren could soon face a new test of how well they're learning music. The test is being developed by two statewide groups of music teachers who see it as a way of reinforcing the importance of music in a well-rounded education and measuring how well it is being taught around Florida.

"Music educators are accountable every time their students step up on the stage, but they felt they needed a more formal way (to measure their learning)," said Timothy Brophy, a University of Florida assistant professor of music education, in a telephone interview from Gainesville...

The first phase of the music tests could begin in fourth and eighth grades as soon as 2007 if all goes as planned. It will consist of a paper-and-pencil test in which pupils respond to a series of questions recorded on a compact disc, including musical passages.

Brophy said the next two phases of the testing program are expected to include actual musical performances that would be recorded and compared with a standard for the appropriate grade level.

Tests for music students in other parts of the country should be modified to conform to local standards and tastes. For example, band geeks in South Dakota should be tested on their awareness of just how seductive the saxophone can be.

"The Year of the Tests"

Testing marches onwards, as 23 states expand their testing programs:

Forty-eight states and the District of Columbia will give standards-based tests in reading and mathematics in grades 3-8 and at least once in high school this school year, as required by the nearly 4-year-old federal law, according to a survey by the Editorial Projects in Education Research Center.

The holdouts are Iowa and Nebraska. Districts in Iowa give the Iowa Tests of Basic Skills, a national test not designed to measure state or local content standards, while districts in Nebraska craft their own tests, except for a state writing exam.

In devising the new tests, most states have defied predictions and chosen to go beyond multiple-choice items, by including questions that ask students to construct their own responses.

Hoo boy. Could be good, could be very problematic (and expensive) to score. The entire article is worth a read.

Are the bad getting better while the better get worse?

Opinion Journal's Best of the Web wonders if there's been an interesting spin on test scores in Michigan. Here's the bad news:

Michigan African-American fourth- and eighth-graders scored much worse in reading and math than African-American students in the United States as a whole, according to national test results released Wednesday. The 2005 National Assessment of Educational Progress results show significant improvement in math. But the test results show little improvement in reading scores overall since 1992.

Now, the good news:

The good news is that the gap between black and white students' scores in math and reading within Michigan has decreased.

BotW assumes that the gap could not have closed without white students performing worse, and this article (interestingly) does not mention white student performance. But it does say that black students have improved in math and stayed level in reading. The "reverse Lake Wobegon" effect might be happening for reading, but not necessarily for math. If the black students have improved in math while the white students stayed even, and stayed level in reading while the white students declined slightly, we'd see this pattern.

I couldn't find change information by ethnic group and state in the 2005 report, but if you manage to see that anywhere, let me know.

Meet the new GRE, same as the old GRE

The GRE, which made waves in 1993 by transforming into a computer-adaptive exam, is being revamped again:

Although the test will still include sections on verbal reasoning, quantitative reasoning and analytical writing, every section is being revised, and the test lengthened to about four hours, from two and a half hours. About 500,000 students, 20 percent to 25 percent of them foreigners, take the general G.R.E. each year. E.T.S., which administers the test, also offers subject-matter tests in such fields as biology, mathematics and physics, but those tests, taken by far fewer students, are not being changed.

To enhance security, every question on the new exams will be used only once, and the test will start at different times in different time zones, so students who have finished cannot pass on questions to those in different zones...

As of next year, the test will no longer be "computer adaptive," with test-takers getting questions tailored to their performance on previous questions, so that each gets challenging questions that provide a clear picture of what they can do. Instead, every student taking the test on a particular day will get the same questions, and those questions will not be reused.

Computer-adaptive tests, as ETS and others have discovered, require item reuse and enormous item pools to prevent any one item from being exposed too often. Good GRE items are not cheap nor easy to come by, and this change addresses security questions and helps to ensure the relationship between the items and the construct by reverting back to the one-use-per-item model. Of course, removing the adaptive algorithm also involves lengthening the exam, because the range in item difficulties for each form once again must be wide enough to adequately assess the geniuses and those who should probably strike graduate school off their "to-do" lists.

Update: Ah, the joys of knowing so many experts. I think this post was up for about two seconds when another psychometrician emailed me to remind me about the CAT version of the ASVAB, the Armed Services Vocational Aptitude Battery. That actually came out before the GRE-CAT, in the late 1980's, and I certainly should have remembered that, considering the truly phenomenal and exhaustive primer that exists on the subject, Computerized Adaptive Testing: From Inquiry to Operation. I can't recommend that book highly enough for anyone who wants to learn more about developing, testing, and implementing a computer-adaptive test.

The MCAT becomes the MCATCAT

The advent of computer-based testing is worrisome to would-be med students:

As if pre-meds did not already have enough to worry about, recent changes to the MCAT have some pre-med students worried about more than just mastering its content.

The Association of American Medical Colleges announced recently in a press release that it will convert the MCAT to a computer-based format within the next two years, a move that will force both students and test-prep companies to sharpen their strategies for tackling the test, rather than their pencils.

The paper format of the test will be administered through 2006, though trial versions of the computer-based test will be given at the August 2006 testing date.

There are many advantages to computer-based tests; three of the biggest ones are the shorter test length (I'm assuming here the new MCAT is adaptive as well as computerized), more testing opportunities, and shorter score report turnaround time. Based on my moderate experience with CBTs, I think that some of the fears of future examinees will turn out to be unfounded:

"With the computer-based test you can't underline passages and put notes next to text, so it's hard to map out the progression of a passage. You can't keep track of it when you have to keep switching from a computer to your notes," said Patrick Wiita, a fourth-year microbiology, immunology and molecular genetics student who has taken the MCAT.

Underlining text onscreen is a fairly simple feature to add, if it isn't already included. And it's also possible to allow a typing area on-screen for notes or comments, for those who don't feel comfortable using scratch paper.

There's also concern about computer glitches, Wiita said. "Who knows what errors could occur in programming. It's the same reason they haven't switched to online voting. If there's an error, there's no paper trail," Wiita said.

Why on earth would you want a paper trail when you can capture every keystroke? Certainly, big programming errors can occur - but paper tests can vanish in transit just as easily. If anything, the software that processes computerized testing errors tend to provide more information to the testing companies than paper errors. There will be exact records of when screens go down, where examinees were when the error occurred - and most CBT providers that I've seen have little problem restarting a test on the correct screen after an error occurs. Any testing company thinking about CBT administrations correctly places a lot of focus on error prevention and recovery.

Mustafa said new testing formats create a lot of work for test-prep companies, especially research into how students feel about the test. "We did a survey on 4,000 (students) to see how they're feeling, what they think. Eighty-two percent said they would do worse on a computer-based test," Mustafa said.

I've yet to see any research showing that examinees in fact do worse as a whole on computer-based tests. Some subgroups, in fact, do better in certain construct areas. Regardless, this is an easy enough question to answer; matched examinee groups can be compared with respect to P&P and CBT scores.

Of course, examinees may experience a feeling of doing worse if they're switching from a P&P test to an computer-adaptive test (or CAT), since the items will be tailored to their ability level and they'll see more of what they consider to be hard items. But CAT scoring scheme takes item difficulty into account, so that two examinees could miss the same number of items but end up with different scores, based on the difficulty of items that were answered correctly.

CATs have been operational in a high-stakes, large-scale environment since 1993 (when ETS pioneered the GRE-CAT), so there's quite a bit of theoretical and operational research out there to guide the development and refinement of the MCAT-CAT. Good luck to them.

Where's the miracle here?

An op-ed in the New York Sun and the folks at The Daily Howler focus on a recent article praising Wake County (NC) for raising test scores. The real picture, it seems, is not what the NYT the made it out to be.

Here's what the NYT printed:

Over the last decade, black and Hispanic students here in Wake County have made such dramatic strides in standardized reading and math tests that it has caught the attention of education experts around the country.

The main reason for the students' dramatic improvement, say officials and parents in the county, which includes Raleigh and its sprawling suburbs, is that the district has made a concerted effort to integrate the schools economically.

Here's what the Daily Howler concluded, after a spot of online digging:

Wow! Times readers felt a familiar glow; 80 percent of Wake County black kids scored at grade level on last spring’s tests! But here’s what Finder didn’t tell you—across the state of North Carolina, 77 percent of all black kids scored at grade level on those same tests! That’s right; the Times devoted this front-page story to a three-point difference in passing rates—a three-point difference in passing rates on tests almost everyone passes!...

WHY YOU’RE BEING PLAYED THIS WAY: Finder’s piece has an obvious sub-text. Wake County is busing to achieve economic integration—and this is producing big score gains.

For ourselves, we would favor such a program as long as the voters were willing. But these Wake Country test scores provide little evidence of big pay-offs in minority achievement if you enact such a program. Yes, Wake has shown good score gains (most likely on easier tests)—but so have schools all over the state! How can Wake’s program account for gains which are happening in all the state’s districts?

Yes, the gains are occurring all over the state...But apparently, Finder didn’t want you to know that. As good pseudo-liberals have endlessly done, he just wanted you feeling real good about a type of program he favors. As good pseudo-liberals have shamelessly done, he wanted you thinking something bogus and cruel: When it comes to the education of poor black children, success is right there for the taking...

And here's what the op-ed writer concluded:

Intrigued by the story's claim that the percentage of Raleigh's students achieving proficiency had risen dramatically over the past several years, my research assistant, Mark Linnen, took it upon himself to check out the data available on the North Carolina Web site. Over the past 10 years, the percentage proficient or better in grades 3-8 in Raleigh (Wade County) had in fact risen by 13% in math and 12% in reading between 1995 and 2005. That seemed to confirm the bragging of local officials - until it was discovered that, statewide, proficiency rates were up by 21% in math and 19% in reading - gains that outstripped those in Raleigh by over 50%. Nor did the proficiency rates of Raleigh's black and Hispanic students climb any faster than the statewide average for these groups. In fact, the gains were somewhat smaller.

Not that proficiency rates in North Carolina mean much. The state has some of the worst state standards in the country. Last spring, my Education Next co-editor, Rick Hess and I gave North Carolina's proficiency standards one of the worst marks in the country - a D minus. (By comparison, South Carolina got an A.) So low were the standards that 85% of all North Carolina eighth graders was said to be proficient in reading, despite the fact that only 29% of the state's eighth graders was found proficient on the National Assessment of Educational Progress, the nation's report card.

One can only suspect that the allegedly astronomical gains in North Carolina - in both Raleigh and elsewhere - were simply a function of a dumbed-down scoring system.

Does the NYT even realize that all the NC scores are online, and that readers can do the math for themselves?

Parapundit has a few things to say on the topic as well

Time to brush up on 10th-grade math

How well would you do on the FCAT? Come Wednesday, you can find out:

State officials will put an old edition of the Florida Comprehensive Assessment Test online so people can see what it's like to be in 10th grade.

If you don't know much about history, don't know much biology, you might still do well. The test measures ability in reading and math. The questions were from the 2004 exam, and most likely won't be seen on any test again any time soon.

State education officials said Friday the test will be on the department's Web site. People can download or print the test and take it like students would, then get the answer key to see what they got right and wrong.

Opening the GATE a bit too wide

The Lodi Unified school district (CA) is unhappy that membership in the gifted-student program Gifted and Talented Education (GATE) is down, and doesn't seem to reflect the ethnic diversity of the local population. So, of course, they've suggested tinkering with the admission formulas:

Trustees recently approved changing the requirements of the Gifted and Talented Education program to allow more opportunity for minority students to join its ranks.

Students are now eligible to receive extra points on top of their standardized test scores for limiting factors present in their lives. So, for example, a child who has a learning disability, is an English language learner or comes from an impoverished or culturally diverse background will receive special consideration.

"We've been seeing lately our (GATE) enrollment is down," Lodi Unified Superintendent Bill Huyett said Monday. "That's an indicator that it needs to be opened up a little"...

Despite past attempts to diversify its gifted programs, the district found GATE enrollment did not reflect the makeup of the district.

Imagine that. Schools tend to find that academic grades, test scores, and other measures of academic achievement are not randomly dispersed among ethnic group members. Seen in that light, the fact that GATE participation doesn't mirror the "diversity" of the district is a validation of the GATE admissions criteria. Unfortunately, "diversity" trumps all, so these educators would like to muddy the waters by including some very fuzzy criteria.

Who's going to define what a learning disability is? What criteria will be used? How severe does the disability need to be? Why would the district expect that a student with any sort of learning disability would be able to handle the GATE curriculum? How long can a student be living in the US and still be considered an English language learner? How are we defining an impoverished background? And why would any of these factors, which could be considered a disability, be lumped in together with being from a "culturally diverse" background, which is not? Does this mean Asian and Indian children who are already off the top of the testing charts will be even more likely to be accepted? Or will some cultures be considered more diverse than others?

I agree that the codes are in conflict; it's ridiculous to define a program as being for kids who are on the top end of the intelligence scale and then also demand that the kids in that group be diverse in any sort of multicultural way. They will be who they are, and if the district is truly unhappy to find out that any particular group seems underrepresented, the solution is to investigate why that might be, not to fudge the numbers afterward with some ill-defined and highly-unreliable admission criteria.

What really frosts my shorts about this whole scenario is that the programs are for the benefit of the students enrolled in them. Therefore, the adults involved should be committed to developing admission criteria that guarantee, as much as possible, that any particular student is ready for more challenges, will benefit by more challenges, and will complete the program with the increased self-esteem that comes from tackling, and overcoming, tough educational programs. The current criteria address this by requiring that students perform above a certain percentile on the Raven Progressive Matrices or the CAT-6 exam.

Instead, the adults involved are dithering over the fact that the "diversity" is not what it should be, and have suggested criteria that may in fact be negatively related to the ability of any particular student to do this work or to benefit from it. Does that sound like a plan that's in the students' best interests to you?

The California Association for the Gifted, or CAG defines gifted students as:

...a child enrolled in a public elementary or secondary school of this state who is identified as possessing demonstrated or potential abilities that give evidence of high performance capability...

"Abilities that give evidence of high performance capability." Period. There is no research supporting the notion that overcoming learning disabilities, or learning English late in life, or coming from an impoverished background, are positively related to high academic ability. Therefore, lowering the academic standards for these students makes no sense. If a child shows that he or she is capable of doing the work, these factors should not be used to exclude them. I see no reason why they should be used to include them, either.

Cutting down on testing in NYC

For schoolchildren in NYC, their number 2 pencils will now last twice as long:

To the relief of thousands of pencil-biting children and their parents, state and local education officials have reached an agreement that means that New York City's third, fifth and seventh graders will have to take only one round of standardized tests this school year.

A conflict between the state and the city had raised the likelihood that children in those grades would have to take four tests over the course of the year - two reading and two mathematics exams - with one set of results for the city and the other set for the state.

Under the terms of the agreement reached yesterday, students will take only the two state tests.

As one educator was quoted as saying, "Common sense prevails."

Do extracurricular activities improve test scores?

New Hampshire wonders which came first - the extracurricular activities, or the high scores?

A questionnaire given to 10th grade students taking part in this past spring's New Hampshire Education Improvement and Assessment Program testing also shows that students who work limited hours while attending classes also have a firmer grasp on subjects...

This year's 10th graders were offered a questionnaire that sheds lights on how activity outside the classroom might impact learning...a statewide analysis of results showed that student who took part in five or more extracurricular activities (i.e. sports, band, theater and more) had the highest mean scale score on this years test.

Students who took part in five or more such activities — of which Laconia had 9 percent responding — had a statewide mean scale score of 269 out of 300 points in reading and 270 out of 300 in math...The questionnaire showed that students who took part in no extracurricular activities...[had mean scores that] place [those] students in a category that identifies them as having a "basic" knowledge of those subjects.

The polling data showed in general that the more extracurricular activities a student took part in, the higher they scored....One difficult thing to determine is whether students are performing better because they are involved in such activities or whether students who are more committed to school are more likely to engage themselves in endeavors outside the classroom...

Yes, indeedy, it is a difficult thing to determine. It's not a surprise that students who work fewer hours, read more outside of class, and take part in more outside activities have higher test scores, but it isn't simple to disentangle these and draw firm conclusions about what causes what.

For example, students could be urged to work less outside of school, which would leave them more time for reading and other activities. But students from low-income families might be forced to work many hours, and it might be the genetics, family dynamics, home environment, and/or lack of parental education that drive the low test scores more than the time spent behind the counter at Burger King. For such a student, the familial incentive to read (or join the marching band) might be nil, while the time spent as a cashier might be helping them learn valuable job and cognitive skills as well as keeping them from unproductive extracurricular activities.

But there was one survey result that suggests a pretty clear cause-and-effect relationship (if I understand the tortured phrasing correctly):

Students who responded as receiving regular homework assignments also showed to perform considerably better on the math portion of the test than those who did not.

Technology to the rescue

A long and informative article about the rise of technological inventions to meet the need of disabled examinees. There's also a nice discussion of the pitfalls inherent when technological modifications could cause a test to be measuring something other than its intended construct.

Another song and dance about the SAT

Do cheesy songs help raise SAT scores? (Free registration required.)

Renee Mazer is trying to help high school students get into good colleges — by teaching them silly songs and cheesy poems.

Mazer is the creator of "Not Too Scary Vocabulary!: For the SAT and Other Standardized Tests and Success in Life," a boxed set of CDs (or audio tapes) aimed at beefing up students' semantic skills. Using playful mnemonic devices and slang-studded stories, the discs teach hundreds of words that often appear on the Scholastic Aptitude Test. By presenting the words in a manner that's easy to absorb and remember, Mazer said, she can help raise students' scores on the verbal section of the SAT. And that can help turn a hapless Ivy League reject into an ebullient Harvard freshman.

What's Mazer's secret? She's never dull.

I'll say, considering that one of the poems talks about emotions involved with "getting to first base."

Raising test scores with a (cinnamon) twist

If you feed them, they will improve:

A pilot program intended to ensure every child has the chance to eat a free and nutritious breakfast is being offered this year to students at Morris Elementary School in Rialto. The program is based on research that shows proper nutrition can positively influence academic achievement, Rialto Unified School District officials said.

The free breakfast program paid for by reimbursable state and federal funds is being offered initially at Morris, but if successful, will be expanded to other school sites, said Syeda Jafri, district spokeswoman.

"We offered free breakfast during standardized testing at our schools during last year,' said Sharon Flores, director of nutrition services for the district. "What we found was that the free breakfast reduced tardies and that we had less students in the nurses' offices with stomach aches.'

This part, though, makes me think teachers will be driven nuts:

"The food is served in the classroom right when class starts so it's a family-type environment,' Flores said.

Which means teachers will have to monitor table manners and clean up all that jelly off the desks afterwards.

A love of tests

A thoughtful and quirky look at the testing craze, from a young (class of '98) reporter:

...I just took last year’s standardized English Language-Arts Test for the 11th grade - which anyone can sample at the state Department of Education’s Web site and which the state uses to monitor its districts’ progress - and while I scored well (a 95 percent, thank you very much), there were only 38 questions, nine of which were devoted to understanding a car rental agreement and the instructions of a food processor. Only one great literary work was examined - “Young Goodman Brown” by Nathaniel Hawthorne - and there were scant vocabulary questions, which, maybe, is a good thing.

...I can say that my foray into the world of standardized testing didn’t exactly fill me with the overwhelming confidence that the results of these tests will mean anything significant. Sure, the state will use them to decide which school districts are doing their job and how to parcel out an increasingly limited chunk of resources, but is it more than just a numbers game?

In speaking with the district’s curriculum director, Elizabeth Chapin-Pinotti, about the county’s California High School Exit Exam and Standardized Testing and Reporting results, which were released last week, it occurred to me that the above question doesn’t simply have a ‘yes’ or ‘no’ answer.

Chapin-Pinotti said that California schools tend to get a bad rap on the national stage when the test results come out, because what’s not factored in is that the state employs some of the strictest standards in the country. “The state set the bar very high and we’re not backing down,” Chapin-Pinotti said.

Reporter Raheem Hosseini goes on to discuss the Connecticut lawsuit before wondering why we care so much:

So what is it about tests we all love so much (and don’t kid yourself, we’re obsessed with them)? I think it has something to do with the simplicity of being able to quantify an amorphous concept such as intelligence. Color in a few bubbles, feed your sheet into the Scantron machine and find out how smart you are. And the strange thing is that even after high school and college and after having taken hundreds of scholastic tests through roughly 20 years of school, we’re not done with them.

There are employee evaluations and credit assessments and loan applications and - even weirder - the tests we actually choose to take in our spare time: crossword puzzles, jumbles, online IQ and personality tests. It never stops.

But is it really so bad for people to crave a little disposable evaluation? Maybe what they’re really craving is intellectual stimulation. Heck, even Tommy Lee is back in school.

Trading one acronym for another

The Chicago public schools are abandoning the Iowa Test of Basic Skills, or ITBS, in favor of a revised Illinois Standards Achievement Tests, or ISAT:

It's welcome news for both Chicago teachers and students who had to spend hours of preparation for the tests and lost precious class time for other lessons. Now, the public schools will be able to focus on one high-stakes test, the revamped Illinois Standards Achievement Tests, which has become an important measure of Chicago schools' performance under the No Child Left Behind law. "It's fabulous," Nobel School Principal Mirna Diaz Ortiz says. "I'm very happy that we're going to be measured by one test and not have to take two tests"...

In addition to the ISAT, there will be another new test, but it will not put the same burden on students and teachers that the Iowa test did. It will not be used to determine promotion. The Stanford Learning First measures students' strengths and weaknesses in reading. It will provide a diagnostic tool for teachers to see where their students need help. It will be offered three times a year through 40- minute exams and will cover the same kind of material that is in the ISATs. It won't require extensive preparation.

The Illinois Standards Achievement Test (ISAT) measures individual student achievement relative to the Illinois Learning Standards. The ITBS, on the other hand, gives results in relation to an Iowa norm, or a national norm. This means that promotional judgments will be made for scores based on comparisons within Illinois, not for the nation as a whole. This sounds consistent with NCLB regulations.

Interactive ISAT samples can be found here. I'd like to be objective, but I note that the eighth-grade Reading sample is a reading passage written by James Thurber, one of my all-time favorite authors. I started reading him about that time, too. So I'm predisposed to like this exam.

This Chicago Tribune article has more info:

The Iowas, used in Chicago since 1972, will be replaced with three short reading assessments that officials believe will prove more valuable in gauging the progress of individual students. Called Stanford Learning First, the new 40-minute exams will be given in October, January and May and will test the same kind of material covered by the new ISATs...Officials said they still expect to use the historic data as a basis for comparison by creating a new formula that can equate the results of the old Iowas with the new Stanfords.

This can work, if the new Stanford exams have the same content as the old Iowa exams. Seems like this is a new, low-stakes way to test reading, with the scores being equated back to the high-stakes ITBS.

The test change also will trigger changes in the district's controversial retention policy, the details of which will be announced in October. The get-tough retention policy was created in 1997 when Daley declared an end to social promotion and started requiring students to meet minimum test standards in reading and math.

The policy has been softened over the years. Now the district only considers reading scores and bars schools from retaining students twice in the same grade regardless of how low they score on the tests. Last year, about a third of the 24,000 students required to attend summer school because of low Iowa scores had to repeat a grade.

The article also mentions that the ISAT-based promotion decisions will be based on the ISAT portion that is scored in comparison to national norms. Interesting.

If any of you have experience with the ISATs, or have information that hasn't made it into the Chicago papers, let me know.

Learning the rules of the road

One of the testing criticisms I often see is the claim that standardized tests can't tell you everything about a student's achievement:

Naysayers to the testing format say it reflects not what a student has grasped conceptually about a subject, but how well they take tests. These critics, most often teachers, point out that each student processes learning differently - some are better able to express verbally or in essay form their depth of knowledge on a subject...

"Think of the driving test," said San Luis Obispo High School's English department chair, Ivan Simon. "If you just looked at how well someone answered the written part of the driver's test, then you'd assume the skill of the driver was represented by only that score. But that person wouldn't necessarily be a good driver."

It is true that the performance-based exam of driving skills tells you much more than the written exam, because the skill being measured is wholly performance. However, one could argue that someone who knows how to turn the key and step on the gas is not a good driver if they cannot read road signs or haven't memorized any of the rules of the road. Someone who passes the written exam is not necessarily a good driver, but we can argue that someone who flunks the written exams is necessarily a bad driver. Both components of the exam are important. The ability to understand signs could be folded into the performance assessment (and often is a part of it), but the reason for the written exam parallels the reason for many standardized multiple-choice assessments - it's a way to very quickly sample a broad domain and make a cheap, reliable assessment in order to flag those who just aren't getting it.

Many standardardized exams are, in fact, minimum competency exams, and the best precision is not in separating the brilliant from the good, but the terrible from everybody else. Simon's criticism that multiple-choice exams often don't tell the whole picture is correct. However, the critics gloss over that these exams can tell you quite a bit about how students have mastered basic skills.

And, as one principal points out, the basic skills are important:

We provide teachers with examples of multiple choice questions, but it's not the sole focus of the curriculum," said [Will Jones, principal of San Luis Obispo High School]. "The state releases sample questions, roughly 20 percent of the test, and if teachers want to use those questions they can, and they do, just like they do for advance placement tests. The misperception is that we spend all our time just teaching to the test and somehow the STAR exam is consuming education, but the truth is there are all kinds of assessments and we prepare kids for all of those as well."

Jones said he believes good students will always excel regardless of test format. "If a student is capable of writing a good essay and answering short questions, then they will have success on the test," said Jones.

This is something that few testing critics are willing to admit. From what I can tell, the evidence that there are hordes of little geniuses out there who routinely flunk exams yet learn brilliantly through non-traditional methods is anecdotal, at best. The testing critics are right when they say that there are skills we aren't measuring with the one-size-fits-all exams, but I say they're missing the boat when they insist that students who can't master basic material are somehow ready for advanced performance assessments.

Who's meeting what standard?

Devoted Reader Lori M sent a provocative column my way:

Tens of thousands of parents of schoolchildren and hundreds of thousands of other taxpayers learned from media reports last week that "the majority of Texas school districts and campuses in 2005 earned the rating of 'Academically Acceptable.'" Most of the moms, dads and school-tax payers breathed a sigh of relief and shrugged off the "bad" news that a small percentage of districts and campuses "received the lowest rating of 'Academically Unacceptable'"...

To "earn" a rating of "Academically Acceptable" on the 2005 Texas Assessment of Knowledge and Skills test, the students in a school district or at an individual campus had to achieve ...

In reading and English language arts, a passing rate of 50 percent.
In writing ... 50 percent.
In social studies ... 50 percent.
In math ... 35 percent.
In science ... 25 percent.

In case the educational horror of those numbers didn't sink in ...
If half of the youngsters in a district or on a campus failed tests in reading, writing and social studies ... and 65 percent failed arithmetic ... and 75 percent failed science — the Texas education establishment deemed that district/campus "Academically Acceptable"!

Lori comments: "I guess I'm part of that old-fashioned school of thought that believes that a passing grade consisted of mastering the majority (at least 65%) of the material. Looks like it's sufficient to learn 25-50%!"

What's happening here? A disconnect between how we, the educational consumers, think of "passing," and how Texas is ranking schools, which is with a minimum-competency standard. "Acceptable" here is not defined in the same way that we'd consider "acceptable" to be in an academic course.

The author isn't exaggerating when he quotes the low percentages above; those come straight out of the state's 2005 Accountability Manual. You can skip right to this table for the good stuff. Yes, it's true that this year, a school for which 26% of the students meet the standard in Science is acceptable in that content area. This is also the first year the Science standard was set at what the advisory panel actually recommended, as opposed to one (2004) or two (2003) SEMs beow it.

To interpret these numbers, you really have to have some idea of what the standards are, so that you know if a school in which only 26% meet those standards is a travesty, or just plain mediocre. You also have to realize that while the state set those standards very low, that doesn't necessarily mean most schools are squeaking in just over the bar. The 10th grade Science results, for instance, show that 54% of the overall student body in Texas met the standard. The raw score conversion table for that exam shows that a raw score of 34 out of 55 converts to the lowest possible passing score, and according to this document, that means a student would have to answer 62% of the items correctly to meet the standard. That's pretty much in line with what most people think of as a passing score.

I don't mean to suggest that parents don't have a right to wonder why the standards for Acceptable schools aren't set higher. And, given that we don't know how difficult the science items are, for example, we don't know how meaningful that 62%-correct standard is. But it might help in this debate to be sure to separate the standard for the exam from the standard for the schools.

Motivation vs. ability

Joanne Jacobs points out that our students may not be dumb - they just might not care:

You could conclude from these exams that American high-schoolers are ill-taught and ill-prepared for the competitive global economy. But what if you look at these tests like a capitalist rather than an educator? Nothing is at stake for kids when they take the international exams and the NAEP. Students don't even learn how they scored. And that probably affects their performance. American teenagers, in other words, may not be stupid. It could be that when they have nothing to gain (or lose), they're lazy...

The dubiousness of these test results becomes clear when you compare them to the results of tests that actually do matter for teenagers: high-school exit exams and college boards...

Alexander Russo, for one, is suspicious of such a neat-and-easy conclusion:

if things are better now in secondary education than they were before, shouldn't kids today still outscore kids from 30 years ago? They were unmotivated to perform on the NAEP then. They're unmotivated now. They know more now, according to Starr. But the scores aren't much different.

And what about elementary school NAEP scores, which are on the rise? If motivation is all, then shouldn't they stay flat?

Now I'm no economist or behavior expert, but it seems to me that if high school kids were actually learning more in school than they had before, the NAEP scores would show at least part of that change.

I'll play Devil's Advocate - could it be possible that kids know more, but care even less? After all, we're constantly told that kids are over-tested and are sick of exams, and perhaps there's truth to that. Could it be that years ago, even though kids knew the NAEP didn't count, they were more motivated, or less burnt-out?

I have no doubt motivation plays some part in test scores, which is one reason that I keep griping at states to stand behind the stakes for their exit exams. But I don't think the explanation is as easy as saying that our kids don't really care. Presumably, they care when they get to college, and yet the rise in remedial coursework and grade inflation belies the notion that today's students, as a group, have what it takes.

Update: Chris Correa notes:

...some research suggests high school students don’t perform significantly better on these tests when they feel more concerned in their own performance. For example, when researchers offered up to $10 per correct question on the TIMSS test, the paid participants did not perform significantly better than the control group. Students did report increased motivation when they were earning an average of $100 for their performance, but this did not make them any smarter.

High school students seem to do about as well as they can on the test, even when they don’t have anything to gain from it.

Excellent point, Chris.

I earned my college money by passing a test

The merit pay discussion takes on a weird twist:

A San Joaquin Valley teacher hopes money will motivate California students to improve their standardized test scores.

Jo Aldrich-Fallert of Porterville will soon start gathering signatures for an initiative that would give $1,000 to parents of children who score at the proficient level or higher on all sections of the California Standards Test. She says she wants to reward families who have made education a priority in their homes.

If the idea proves to be a successful motivator, the state could end up paying out more than $1 billion to families of students who do well on the test.

Critics of the idea say doing well in school should be its own reward. They also worry about the cost at a time when the state is strapped for money.

Seeing as how money doesn't grow on trees, I agree with the critics here. I hope this idea gets more publicity, though, just for the amusing debates it will generate. If we're willing to pay parents, but not teachers, more money for higher test scores, aren't we admitting that parents have more to do with a student's academic achievement than the teachers? Isn't that support for homeschooling? And what about the fabulous, motivated parents whose children get stuck with turkeys for teachers? They'll miss out on the dough because a hapless educator undid all the hard work of tutoring and motivation that was done at home. If this amount of money was on the line, wouldn't parents be even more observant - and critical - of bad teaching and bad schools?

Like I say, amusing.

It's all grist for the testing mill

It's an ironic world we live in:

UNCG professor Svi Shapiro has written articles critical of standardized testing. Now, he's become part of the test. One of four prompts used for June's essay section of the newly revamped SAT was adapted from Shapiro's writing.

"I became the test instead of criticizing the test," said Shapiro, who has taught at UNCG for about 20 years. "It is ironic."

Shapiro learned that an adaptation of his work had been used on the test while giving a lecture at the Governor's School of North Carolina. The prompt asked test takers to explore whether schools should help students "understand moral choices and social issues." "I was surprised, to say the least," Shapiro said. But he said the question captured the essence of his criticism of standardized testing...

Raise test scores, make more money

Talk about your merit pay:

Indianapolis Public Schools may be among the first in Indiana to link their superintendent's pay to student success, but a growing number of urban districts across the country already have been taking that approach. School boards in Cincinnati, Dallas, Denver and Illinois have all negotiated contracts with their top administrators based on meeting tough academic goals.

The Palm Beach County, Fla., superintendent can earn a 15 percent bonus on his annual $155,279 salary if test scores dramatically improve, suspensions drop and graduation rates go up.

That's...a lot to hang on one person, especially a person who isn't in the classroom. On the other hand, if he actually has the power to change things, well, this is a pretty good opportunity to do so.

A psychometrician's worst nightmare

Normally, when I read an newspaper article about a student who's been struggling with a standardized test and keeps coming in just under the wire, I'm pretty unemotional. Yes, that sucks, but a cutscore's a cutscore. While the setting of the cutscore is a delicate process, there will always be someone who just missed out no matter how it's set, and I'm usually willing to defend the process against that unlucky fellow who's just a point or two too low.

But I really don't want this guy mad at me:

All that stands between Garry Williams and a spot on the University of Kentucky's football team is a point on a standardized test. And it makes Williams downright mad. Not necessarily mad about the test or the NCAA's entrance requirements that call for student-athletes to achieve a minimum grade-point average and test score. Just ticked that he's not playing football at UK.

Williams could be entering his sophomore season at UK and likely competing for a starting job on the Wildcats' offensive line. Based on the billing he earned as a first-team All-State performer at Seneca High School, Williams might even be a returning starter.

But the NCAA no longer allows "partial qualifiers" -- athletes who meet only one of the two minimums for grade-point average and standardized test score...

The article emphasizes that neither Williams nor his father used this opportunity to bash the test (in this case, the ACT), which is admirable. But Williams is mad, and who could blame him? Who's to say what might happen should he get more and more frustrated with the studying, and then one day happen to (gulp!) meet a psychometrician in the street?

Full disclosure, Williams - I don't work for ACT. Now put down the barbell. Please?

Knowing how badly he's needed only adds to the frustration...Garry Williams Sr. said. "To be honest, he's ready to strap on the pads and hit somebody."

Aaaaaah!

The gap narrows

More good news from the elementary schools:

Black and Hispanic students are catching up with their white counterparts in reading and math at the elementary-school level, but there has been little closing of that achievement gap in higher grades, according to a study released yesterday.

The Bush administration cited the data from the National Assessment of Educational Progress (NAEP) as evidence that its educational revisions are working. But the independent body that administers the tests urged caution, saying that many of the gains could have come from changes made before the 2002 passage of the No Child Left Behind Act.

The NAEP study of long-term educational trends showed a significant improvement among white, black and Hispanic 9-year-olds in the 2003-2004 school year in math and reading, compared with results from five years earlier. But blacks and Hispanics made greater gains than whites in both subjects.

The seriousness of testing

Alfred Orsini spots a link between the "fat" society of the US and the unwillingness to get behind tests with real bite to them:

China is a country that promises to change the economic face of the globe in the years ahead. It has an incredibly lean, mean education machine. American policymakers, seeing embarrassing data such as the poor showing of U.S. students on recent international assessments in science and math, say they want to prepare our children to meet the global competition posed by countries like China.

If the No Child Left Behind law is meant to do that—to help us compete with countries that have used big tests for a long time to scientifically weed and stratify their citizens—the plan will fail. This is not just because of the problems inherent in creating and enacting such tests, but also because of the differing social, economic, and cultural contexts that surround such tests...

U.S. business leaders, in their urgent push to whip American education into line, may be among the few in our country who are truly aware of how things are outside the United States. They have a concrete motivation to be aware: money...But if my perspective from China is valid, then a more general “leaning” of America may have to happen before big tests are widely tolerated by U.S. students and their families.

For now, we Americans can make all the tests we want. Kids will never be “lean and hungry” in a fat society. Big, consequential tests run up against a lot of obstacles in America. One of them is fundamental—what the historian Richard Hofstadter labeled in his influential book as Anti-Intellectualism in American Life. Others include the distraction posed by the cultural excesses that assault American kids every day and the fact that U.S. students know there is a college in America for just about anyone who can pay for it. But beyond such problems, I wonder if American kids sometimes don’t care about study because they sense that many adults don’t care about them—about the life of their minds and the enrichment of their souls.

On the one hand, I can understand why some critics claim that the US is test-obsessed; certainly, those on the front lines of education see it that way. On the other hand, how many of those tests have serious consquences attached? And by "serious," I don't mean "You'll have to go to Penn State instead of Harvard." The great hue and cry from the educrats over the fact that now, for the first time ever, schools are forced to justify their funding by showing student improvement suggests that Orsini is onto something with his theory. Americans are far more enamored of the idea of second chances and alternate pathways than they are of tests with truly high stakes.

When success in English is a problem

From the Interested Participant comes a fascinating tale of teachers complaining about high test scores, and superintendents claiming that it's all about the money:

Thousands of Arizona students struggling to learn English are about to lose extra help because a new state test shows they can read and write in English, though educators fear many of them really can't. In some districts, students are passing a new state test that says they are proficient in English at nearly double the rate of last year. But educators say the test is just easier and these kids aren't likely to pass regular classes without help.

Arizona Superintendent of Public Instruction Tom Horne said the criticism comes because some school districts don't want to lose the extra money, about $350 per English-language learner. He said some schools have kept students designated as English-language learners for years because it means more money.

What a mess. Certainly, though, if the teachers are correct in their suspicions about the test, the results should be pretty much immediately apparent; a doubling of the overall pass rate is certainly cause for concern. However, I'm not sure if they're comparing apples to apples here:

Arizona school districts used to have the choice of one of four language tests, but the federal government now requires states to select one test for consistency. Arizona chose a Harcourt Assessment test, along with several other states...

"We're very concerned," said Cindy Segotta-Jones, director of language acquisition for Cartwright Elementary School District in Phoenix. She said about 2,200 Cartwright students passed the language test, nearly double the usual rate.

But on what test was the baseline set? If 1100 students passed last year, was that over all four tests? If so, is it possible that the other tests were too hard, and this one is closer to being just right? There's no way to tell from this information, of course, but it would help if I knew the pass rate on this exam from last year vs. this year. Now that everyone is using the same, it will be possible to compare results between schools in way that wasn't feasible before.

Sal Gabaldon, a language acquisition specialist for the Tucson Unified School District, said the new test is far different from previous ones used in Tucson schools.

The previous test used in Tucson required students to pass all three parts - reading, writing and oral - to be declared proficient in English, he said. The new test uses a composite score so it's possible that a somewhat lower score in writing could be offset by a higher oral score.

Becuase the previous test was triply conjunctive, one could argue that it was more difficult to pass, even if the individual items were the same difficulty as on this exam. However, is it a bad thing if the test isn't conjunctive? I'm not a content specialist in the area of English proficiency, so I don't know (a) how possible it is to get wildly different scores in the areas of reading, writing, and speaking, or (b) whether having wildly differing scores is predictive of disaster. Is it highly unlikely that a student will be a disaster in one area and great in another? If a student is good enough to pass in any one of the three areas, will they have a decent chance at succeeding later on without additional remediative help? I'd have to know all of those things before I could judge whether the test is useful for this purpose.

Tucson's teachers apparently think these results presage disaster, but Interested Participant wonders if some of that is hurt feelings from having control over exam choice taken away.

Unhappy with MEAP results

In Michigan, sliding MEAP scores are scaring officials, who believe the state's chances of attracting more high-tech companies are in jeopardy:

...a shrinking percentage of high school students are exceeding the state's requirements in math and science. Results from Michigan Educational Assessment Program tests for the class of 2005, released this month, give the state a clearer picture of the challenge it faces in its effort to attract high-tech businesses with a better-educated work force...

Over the past six years, the percentage of high school students scoring well below what the state considers failing on standardized math tests grew from about 20 percent to 29 percent, which translates into almost 32,000 students in the class of 2005. In science, it grew from 20 to 26 percent. Meanwhile, the percentage of students scoring well above state standards fell from 22 percent to 9 percent in math and from 7 percent to 5 percent in science.

Testing experts and school officials caution that there could be many reasons for the trend in math and science other than students' skills getting worse. They say more kids are taking the test to meet federal requirements of the No Child Left Behind Act. Under the law, schools must test 95 percent of their students. These students, who previously may have avoided the test because they felt they wouldn't do well, could be bringing down scores, they say.

Certainly, an influx of underqualified examinees can reduce the percentages - but I'd be interesting in knowing if the raw numbers have changed. If the actual number of students scoring at proficient has stayed the same while the percentage dropped, that would be evidence of the more-underqualified-examinees theory. However, if the number of students who are doing well is dropping along with the percentages, then something else is happening here.

I wouldn't feel comfortable assuming that the numbers above were signs of random, meaningless fluctuation. Either the examinee population is changing, the tests aren't equated very well, or it's some combination of the two (perhaps along with other factors).

The new math means no math for Virginia teachers

Several Devoted Readers have sent in the news that the Virginia Board of Education is phasing out the Praxis I licensure exam for teachers. There is a new exam for these teachers that apparently doesn't contain any of the tricky math that was on the Praxis I:

Praxis I is a basic reading and math skills test that has been a hurdle for some teachers--especially the math portion.

Starting Jan. 1, the state will require teachers to take a new test for licensure. It will eliminate the math material of the Praxis I. Instead, it will require teachers to analyze readings, write an essay, interpret tables and graphs, and demonstrate knowledge of grammar and vocabulary, all "on a college level," said Charles Pyle, spokesman for the Virginia Department of Education.

The Washington Post claims that VA teachers will now have to be more "literate and proficient in the subjects they teach," but those who don't intend to be teaching math need not worry about a standardized math test:

Kate Walsh, president of the Washington-based National Council on Teacher Quality, said she pushed for a new literacy test when she testified before a Virginia panel examining licensure requirements. Research has shown that the ability to read and speak effectively is the most reliable predictor of future success in the classroom, she said...

The Virginia board claims that the this new test actually represents an increase in standards, which makes me wonder - if the pass rate drops dramatically from the Praxis I rates, will Virginia stick by the new exam?

The Cranky Professor is skeptical:

What's not at all clear from the article is what the 'Instead, they will have to pass a new "literacy and communications skills" exam that will be introduced in January' is. Will it be a product of the ETS people, like Praxis? Will it be home-grown, in which case I dread the first 3 or 4 years of results and controversies.

I took the Praxis I a long time ago, by the way, and it really IS at the 8th-10th grade level (I was half-heartedly pursuing certification to teach high school Latin). If people can't pass it (the Post's anecdote has a PE teacher passing on her 6th attempt) they probably aren't capable of figuring their own grades.

Why is no one asking why so many teachers - who are, after all, college graduates - are having so much trouble with basic math skills? The Praxis I math test has only 40 items and takes only an hour - but calculators are not permitted. The sample items mentioned in the ETS link are not tricky, and most of them, quite frankly, could be done in one's head, or with a bit of pencil scribbling.

Do English and music and PE teachers need to know this much math? Not necessarily. I just find it fascinating that schools of education are apparently churning out college graduates who can't do this.

Update: This can't not be related to the observed dumbing-down and politicization of math instruction in schools. Joanne's comments on this article are not to be missed.

Also, Right on the Left Coast has much more to say.. He also quotes the excellent book Innumeracy on the astounding phenomena of how people who would be ashamed to admit they can't read have no problem admitting that they don't understand basic math. It's all about being a "people person" instead of a cold, impersonal number cruncher, don't you know.

Update #2: As always, Joanne cuts right to the point:

Thirty-five years out of high school, I can do these problems in my head. It's hard to believe there are people smart enough to teach who can't pass a basic math test. How are they going to average students' grades?

Why does Joanne assume that these math-phobic teachers will be assigning objective, numerically-based grades? My guess is anyone this terrified of Praxis math items will be giving "holistic" and subjective letter grades.

That's much more "personal" and "caring" than those nasty ol' averages, you know.

A student speaks up in praise of testing

An amusing (because it's positive) editorial on testing from the Arizona college newspaper:

They say the tests are ruining the educational landscape -- that they're causing students to study for nothing else...Many want to see them eliminated. These opponents point to statistics like the amount of money people spend on preparation, $310 million a year by one count, as signs that the standardized-test mania has gone too far and that a return to the core values of education is warranted.

However, such logic fails to understand the purpose that they serve. Simply put, standardized tests serve one vital function – as providers of a means of comparison to admissions panels.

What makes them so valuable? Why can't admissions panels look at more important factors? Factors like grades, letters of recommendation, and extra-curricular activities? That's where the standardized part comes in. Without standardized tests, there is no way to compare students across the country. With GPA, for example, there's no way to tell whether the 3.6 from Northwestern is more impressive than the 3.6 from Virginia.

So standardized tests are good because they're standardized. But rather than complain, UA students should cheer this opportunity as they approach grad school rather than boo it. It's hard enough competing with students from big name schools like MIT and Princeton. With the GRE/MCAT/LSAT/GMAT to level the playing field, students can show that they're just as sharp as the Ivy-League trained type.

In fact, that's why tests of these kinds were developed. With all the anti-testing commentary in the press, it's easy to forget that. What's more, it's always driven me crazy that opponents produce the over-inflated and unproven expensive test prep course as evidence (somehow) that the tests themselves are a problem.

Cute photo of the author, by the way.

Also, this is the nth article I've seen that references the book, "Freakonomics," so I'm going to have to break down and buy a copy.

Portfolios in NY?

The portfolio option arises again, this time as an alternative to the Regents exam ($ubscrip required):

One of the items still on the table in this final week of the legislative session in Albany is a bill that would direct the state education commissioner to come up with an alternative to the Regents exams required for a high school diploma in New York...

The bill has already passed the state Senate and is now before the assembly, where the speaker, Sheldon Silver, can either pass it or kill it. Our friends in the education policy community...are hoping the speaker will kill the bill, seeing it as a departure from the measurement, accountability, and standards that are essential to quality education. The New York Times and the Daily News are in agreement on the point.

I can't find this story anywhere else (the blurb from Google quoted a line about portfolios), and, unfortunately, that's all I can get without subscribing. Anyone out there got a subcription?

Update: A sharp-eyed reader found a more informative article from back when the bill had just passed the Senate:

Supporters say it's better than the state's current one-size-fits-all system of testing. Opponents, including state Education Commissioner Richard Mills, fear the measure could cripple their decade-long effort to use the Regents exams as a way of raising academic standards across the state...

One of the bill's sponsors, Stephen Saland, R-Poughkeepsie, who heads the senate's Education Committee, said the measure stems from what he views as an unwillingness by Mills and the board to compromise on having all students take the exams. "It's their way or the highway," said Saland.

But Mills says setting up a system of portfolios would be too costly and difficult to police. "The bill would create an unworkable system," predicted an Education Department memo to lawmakers. "It would be virtually impossible to monitor all schools closely enough to ensure that schools followed the curriculum and assigned all projects, and that an A in one school equaled an A, not a C, in another school."

When it comes to school accountability, what exactly is wrong with "their way or the highway?" And Mills hits the nail on the head with his concern about the un-standardized nature of portfolio projects.

Update #2: The Instructivist has more, and links to a NYT editorial which is, astonishingly, crictical of the bill:

Before they jeopardize education reform, legislators should revisit a disturbing report issued a few years ago by a panel of education experts that evaluated the portfolio assessments used by the schools in the New York Performance Standard Consortium, a politically influential education group. The panel could find no evidence to support the claim that the consortium's schools were conforming to the state's learning standards or measuring student progress in any meaningful way.

Struggling with algebra

A hullabaloo in Georgia over the dismal algebra end-of-course exam results:

More than two-thirds of the DeKalb County school system's eighth-graders failed the state's End of Course Test in Algebra I, according to preliminary results. The End of Course Test counts for 15 percent of students' final grades in the course. Teachers issued failing grades to 21 percent of all eighth-graders. This was the first year district policy required all eighth-graders to take high school algebra, a move the board approved in 2003 to beef up math instruction in all grades.

Officials reviewed first-semester grades and anticipated a high failure rate. Originally, the policy called for eighth-graders to earn credit toward graduation. But the school board is expected to approve a policy change in July. Parents will get to choose whether their child gets a credit toward graduation, known as a Carnegie Unit, or a middle school credit that does not show up on the transcript college admissions officers see...

For many, it would jeopardize their chances for a lottery-funded HOPE scholarship. Other issues cited by board members: This year's eighth-graders did not have the foundation starting in kindergarten for algebra, and some teachers were not adequately trained to teach algebra.

All pertinent issues. The officials were wise to antipate a poor showing. The question is whether results will improve in the future.

No grade inflation on this scale: State tests measured against NAEP results

Tough state standardized tests - how does your state measure up?

A newly published study...conducted by Paul E. Peterson and Frederick M. Hess, editors of the quarterly journal Education Next...compared how fourth and eighth graders performed on state tests compared with the 2003 National Assessment of Educational Progress test.

State tests are used to rate school performance under the federal No Child Left Behind law. Each state creates its own test and sets the passing score. The result is states with easy tests can appear to be doing well, while those with challenging tests can appear to be doing poorly.

"Some states have risen to the challenge and set demanding proficiency levels for their students, while others have used lower standards to inflate reported performance," the report said. "Not only is the disparity confusing, but, perversely enough, the states with the highest expectations often stand accused of having the most schools said to be in need of improvementeven when their students are doing relatively well."

The top five states in the rankings were South Carolina, Maine, Missouri, Wyoming and Massachusetts.

In other words, if a state claims that fewer students are proficient than NAEP scores would indicate, that state gets a higher score for challenging its students; the opposite holds for states whose students do not seem to perform as well on NAEP as on state exams. The full list of 40 states (10 do not have state tests which allowed for NAEP comparisons) is here.

Mistreating kids and blaming the WASL

School administrators suspend a fourth-grader who balks at a WASL item:

Tyler Stoken is 9 years old and his mother says he's good at taking tests. But when it came to the recent Washington Assessment of Student Learning, one question stumped him. He was asked to write a short essay about a make-believe situation and his principal.

Tyler paraphrases the question saying, "You look out one day at school and see your principal flying by a window. In several paragraphs write what happens next." He's asked, "So why didn't you answer that question?" He says, "I couldn't think of what to write the essay without making fun of the principal."

He refused to answer the question even after his mother was called to the school. Tyler's mother Amy Wolfe says, "And he said he didn't know the answer. He just didn't know what to write. And they were telling me to make him answer the question."

He still didn't, so Tyler was given a 5-day suspension. In the letter that went home to mother, the principal writes, "The fact that Tyler chose to simply refuse to work on the WASL after many reasonable requests is none other than blatant defiance and insubordination."

Amy and her son were shocked. Just then the phone rang. It was the superintendent calling to apologize. "Because I think a mistake was made and over reacting to Tyler's refusal to complete the test," said Aberdeen school superintendent Marty Kay. He says it points to the bigger issue of how much pressure is placed on students and staff to do well in the WASL.

The last I checked, no test in the world required every single examinee to get every answer correct, or even to answer every single item. Refusing to answer one item is not the same thing as refusing to take the test. The school's mistake was appalling, but equally appalling is the "excuse" that the WASL places too much pressure on the staff. This sort of nonsense plays right into the hands of testing critics, as well it should.

This shouldn't turn people against the WASL - it should turn parents against administrators who think badgering kids is the only way to raise WASL scores.

Via Morons.Org.

Teacher resigns over testing law

Devoted Reader Michael S. brought my attention to a story from his local paper:

Eighteen-year veteran Champaign Central High School mathematics teacher Kathleen Smith stunned board members by resigning as a protest, she said, against district and federal policies that force teachers to "teach to the test." During Monday's school board meeting, Smith said she's resigning because she's at odds with current standards in the district and with methods imposed on teachers by George Bush's landmark No Child Left Behind legislation.

"Each year students come to me with different skills, different strengths and different weaknesses," Smith said. "It's always a learning process. They learn about me and I learn about them. Now I find myself constrained by a mentality that says all students will learn the same material at the same pace and prove it by taking the same multiple-choice test within a given time frame. I do not believe a student's understanding of mathematical concepts can be assessed by a multiple-choice test, nor do I believe that such a test is fair for all learners," she said. "I'm resigning because I'm caught in a moral dilemma. "

Fascinating. She decided to resign immediately after Bush was re-elected this past November, but waited to make her announcement at the end of the school year.

Shaky on social studies in NYC

New York City's eighth-graders aren't distinguishing themselves in social studies:

A stunning 81% of the city's eighth-graders flunked the state's basic social studies exam last year - and the scores have gone down annually since the test debuted in 2001. The troubling spiral was disclosed by Education Department officials yesterday at a hearing on civics and social studies instruction called by the City Council's Education Committee.

"Clearly we have a crisis on our hands," said City Councilwoman Eva Moskowitz (D-Manhattan), who was chairwoman of the hearing and blasted educrats for having a lack of urgency about how to adequately address the problem. The failure rate for eighth-graders on a test that measures students' knowledge of basic history and government has climbed steadily from 62% in the 2001-02 school year, to 76% in 2002-03 and 81% in 2003-04.

I think the conflicting reasons given for the decline - that teachers are focusing too much on reading, or that students can't read and thus don't understand the items - are priceless. But then, I don't have a child in this public school system.

So what are these basic history and government items like? I think this is the exam they're talking about (let me know if you know otherwise), and here's one of the June 2004 exams. Forty-five MCQ's and a few open-ended items. I found the scoring key, but I don't know the cutscore, so I don't know where the standard is set.

The MCQ's aren't gimmes. They do assume that students have memorized quite a few facts about Native Americans, American history, presidential elections, US geography, and the like. Take the item below, which is well-constructed and crystal clear - but if a student doesn't know the answer, guessing is their only hope:

The Supreme Court decision in Marbury v. Madison (1803) established the principle of

(1) judicial review
(2) separation of powers
(3) habeas corpus
(4) nullification

Are these questions going to be tough for kids who can't read English? Absolutely! Does that mean the test is too tough for NYC eighth-graders, or that they shouldn't be able to read on this level or answer these types of items? I'd say no, but I'm not a content expert in social studies.

The lifecycles of testing

An amusing and thorough article about the rise and potential fall of standardized tests:

...the consensus is that standardized tests weren't created for such a sweeping, high-stakes purpose. Scores point out a child's strengths and weaknesses, but they don't paint a complete picture, say experts, known as psychometricians.

They cite other problems with high-stakes tests: greater motivation to cheat and the possibility that results will be distorted by overpreparation.

"That's the position of our entire field," said Steve Dunbar, head of Iowa Testing Programs, developer of the Iowa Test of Basic Skills, taken by more than 350,000 Georgia kids in grades three, five and eight. "A test is just a snapshot of where that particular kid is on a particular day"..

Experts in the Lindquist Center, where test booklets are stacked high against the walls, expect No Child Left Behind to run its course. They're confident the politically driven pendulum will swing back to a more reasonable view of the value of testing. Dunbar predicts public support will wane because of results that don't seem to make sense — as when a highly regarded school like Cobb County's Walton High gets dinged for not making enough progress, which happened in 2003. (Like many Georgia high schools, Walton did not test 95 percent of its students, as the law requires.)

"The tests," Dunbar said, "will lose credibility."

Until they do, life is sweet for those schooled in test development. Testing companies, academic think tanks, public policy groups and state agencies compete for the great minds in testing, especially those that come out of the University of Iowa.

In fact, I'd say competition is fierce even for the not-so-great minds, as evidenced by the continuing employment of yours truly (heh). On the other hand, pretty much every testing company is understaffed and overworked, which is definitely "unsweet" in many ways.

Letting kids skip the exams

The question of boycotting state exams is being posed in New York state:

Ryan Ficano, 10, and his fourth-grade classmates at Center Street Elementary School are scheduled to take a state-mandated math test next week. Ryan, who boycotted the state’s English language arts test in February, hasn’t decided whether to take the test, his mother, Carli Ficano, said Thursday night...

Ficano was one of eight parents who met at the Center Street school to talk with administrators and teachers about required testing and the philosophy of mandated exams. Her discussion with Ryan would include teachers’ favorable and neutral opinions of the tests, she said, and that it would be OK not to take the test this year if he chose.

Center Street's principal has asked parents who oppose the testing to take their concerns to the state legislator instead of allowing their kids to boycott (which could certainly hurt the school standing.)

Understanding the numbers

There's an abundance of standardized tests these days, and an equal abundance of confusion from test-wise, results-foolish districts that aren't quite sure how to use all that data. A town-gown collaboration aims to correct that, as Boston school district employees and Harvard faculty and students team up to write the book on using assessments wisely:

Data Wise: A Step-by-Step Guide to Using Assessment Results to Improve Teaching and Learning will be published by the Harvard Education Press in November. All royalties from the book will go to the education school to work with the 58,300-student Boston public schools. The book grew out of a yearlong workshop designed to help the district’s teachers and administrators learn ways of making more productive use of student test results and other data.

Experts say the book’s partnership between researchers and educators could help bridge the gaps that exist between testing and instruction...

At the start of the school year, educators often describe feeling overwhelmed by the amount of data and where to dive in...So the workshop and the forthcoming book are structured around an “improvement cycle,” with the tools to use at each step along the way.

While the process might look different in different schools, the cycle helps educators: identify patterns in data; choose key issues to investigate; dig deeper into multiple data sources; agree on a problem and explore its causes; examine current classroom practices; draw up a plan to change those practices; carry out that plan; and then assess the results of those actions.

Next up: "Dance your way to a higher score"

Schoolhouse Rock meets the SAT:

Pop culture references, jokes about dating and female empowerment messages - "Not Too Scary Vocabulary'' ($45.95) sure isn't your parents' SAT-prep program. Renee Mazer, a University of Pennsylvania Wharton School graduate and a standardized-test tutor for 17 years, uses poems and songs to help students study vocabulary in this seven-CD set. "I watch TV, what kids are watching, listen to the radio. Kids like 'That '70s Show,' and they are listening to the music I'm listening to. All those references are there.'' Some students have told Mazer their scores have jumped 70 to 150 points after using her prep kit.

Putting NY teachers to the test

Hot on the heels of the ridiculous cheating scam one flunking NY teacher tried to pull, Newsday runs an article entitled, "Don't Stop Now In Testing Teachers."

For years, the city employed hordes of teachers who had been in the classrooms for years, even though they never had been able to pass the teacher certification exams. One exam tests the teacher's general knowledge of basic skills such as reading, writing and math, while the other tests the teacher's knowledge of the subject he or she teaches...

Wayne Brightly was able to remain in the classroom for 13 years because he was hired before the new rules took effect and because he got several extensions of his temporary teacher's license. But he hadn't passed the exam in his subject area. His last extension was due to expire in August.

The educators I talked to said the certification reforms have improved the quality of teachers over all, aided by programs such as Teach for America and NYC Teaching Fellows, which have drawn new recruits by making teaching seem a cooler and more professional job. The written exams matter because research shows that students perform better when their teachers have high verbal skills and a mastery of the subjects they teach.

I don't know which specialty exam covers Brightly's subject area, but here's the Liberal Arts and Science test (LAST) is here.

Testing round-up

Despite recent criticism by UC officials, the College Board trustees voted overwhelmingly to continue using the PSAT as a scholarship qualifier. The arguments used by those opposing the PSAT are exemplified here, where one critic makes it clear that he doesn't consider it fair that students from wealthier homes do better on tests.

Apparently, recent NY DOE practice math items were a lesson in spotting errors. Unreadable graphics, misspellings, items with no right answers...not a pretty sight. (The Powerline guys are succinct: "I assume that pretty much all fourth-graders can spell 'fourth.' So who in the world writes this stuff?"

Let me know if you can make heads or tails out of this coverage of an education-related speech by Professor Joel Spring of Queens College. I can't, especially the part about how competition in schools and intellectual freedom are somehow mutually exclusive. (Free reg required.)

In Northview, MI, they're using cool cereal bowls as part of the breakfast of champions, in order to facilitate learning (and improve test scores).

Another op-ed sees a link between increased emphasis on test scores and the Red Lake School massacre. I agree, though, that schools need to help students feel safe emotionally as part of the academic environment.

Opting out in Colorado

In Colorado, the number of "conscientious objectors" against standardized testing is on the rise:

[Brentwood Middle School principal John Diebold] said that 10 of the school's 670 students opted out of this year's standardized tests, more than any previous year. He has to honor the parents' right to choose while dealing with the negative affects the decisions have on his school.

When a student opts out of the test, the school receives a minus score for that test, which affects the "report card" the school receives from the Colorado Department of Education. Diebold said he uses the test results to gauge what they are doing well and what they need to improve.

"When parents have their kids opt out of the test it's hard to get the total picture of what your school's capabilities are because your whole school is not testing," Diebold said. "There are no legal ramifications, but it wouldn't be something I would advise."

Antoinette Medina is one student who is opting out; her mother was apparently influenced by testing critic Don Perl, who claims the Colorado State Assessment Program exams (CSAP) create a gap between the "haves" who do well, and the "have-nots" who don't. It's odd, isn't it, the way the test itself is seen as the stumbling block, as though if the test were removed, there would no longer be a gap between those who learn basic skills and those who don't. All that would vanish with the test's removal would be an objective way of assessing the amount of material students have learned.

As for whether or not the CSAP items measure learning, I've yet to see the test's critics present any evidence suggesting that the test items are not valid for the purpose for which they're being used. There are plenty of released items online if you'd like to check them out for yourselves.

Secretary Spellings speaks out

Secretary of Education Margaret Spellings defends tests in the Hartford Courant:

Connecticut Education Commissioner Betty Sternberg has asked the U.S. Department of Education to exempt half of the state's students from annual testing under the No Child Left Behind Act. She said, "Adding tests in grades 3, 5 and 7 ... will tell us nothing that we do not already know about our students' achievement."

I disagree. For one thing, it will tell you how well your third-, fifth- and seventh-graders are doing. Teachers cannot remedy weaknesses they don't see. The whole point of assessing students regularly is to catch problems early so they can be fixed before it's too late...

Connecticut has received more than $23 million to develop its assessments. Commissioner Sternberg claims the assessments for grades 3, 5 and 7 would cost Connecticut another $41 million. This estimate is off the mark. It includes costs either unrelated to testing, such as "curriculum adjustment" and school choice, or met by the federal government already, such as professional development. The testing mechanisms are in place - they simply need to be applied to the rest of Connecticut's schoolchildren...

Former Clinton administration official and state Education Commissioner Gerald Tirozzi, who now leads the National Association of Secondary School Principals, called it "two Connecticuts: separate and unequal." Students were misdiagnosed, victimized by low expectations and hidden behind district-wide averages - out of sight and left behind.

President Bush saw this for what it was: unacceptable. Today, nearly every state has reported improved academic performance, with minority students and urban schools posting some of the greatest gains. Thanks in part to No Child Left Behind, the pernicious achievement gap is finally beginning to close.

Attacking the PSAT

Not content with picking on the SAT, UC officials are now going after the PSAT as well:

...officials at the UC system have a new target: what they perceive to be the National Merit Scholarship Program’s overdependence on the the SAT’s cousin, the Preliminary SAT. A forthcoming article in National Crosstalk reports that a faculty committee at the university has recommended that the system’s campuses stop awarding National Merit Scholarships, and that a longtime UC administrator at is asking the College Board to break its ties to the program.

At the core of the university’s objections is the belief that by using students’ scores on the PSAT examination as a strict cutoff for whether they qualify as National Merit Semifinalists, the merit scholarship program discriminates against black, Hispanic and American Indian students and students from low-income families who, on average, score significantly lower on standardized tests than do their white, Asian American and more-privileged peers.

Classic misunderstanding of test bias. The results of using PSAT score as a cutoff don't agree with the politically-correct version of how UC officials think the world should work; thus, the test is allegedly unfair.

There is nothing wrong with asking, as some UC officials and former officials are, if the PSAT is a valid test in this situation. It's true that it was not designed (to my knowledge) to be used as a means to obtaining a scholarship. However, it measures basically the same skills as the SAT, and thus it is an early measure of students who should be on the college prep track, and who could most likely use the money. (Full disclosure: I was a National Merit Scholar.)

College Board officials could not be reached for comment on Sunday. But Wayne Camara, the board’s vice president for research and development, told Crosstalk that the process by which the National Merit Scholarship Program and some individual colleges winnow the 16,000 semifinalists down to the 8,200 students who actually receive National Merit Scholarships each year takes factors other than PSAT scores into account. “The practice that National Merit is following is very consistent with the requirement that they use multiple sources of information in making a high-stakes decision,” Camara told Crosstalk.

Camara also said in the interview that because the PSAT has been shown to be valid in predicting students’ SAT scores, the SAT’s validity in predicting student performance extends to the PSAT.

It's nice of the Colloge Board to point that out, but the heart of the complaint here is that not enough of the under-represented groups get scholarships when the PSAT is used for selection. It's disappointing to see the test being attacked in this situation, rather than a school system in which minority students are overwhelmingly short-changed. We can certainly discuss predictive validity and cutoff ranges, but those attacking the test would do well to step back and ask themselves just why so few minority students (and poor students, allegedly) are able to score at the high percentiles required for scholarship selection.

Self-esteem and the SAT

Touchy-feely educators, listen up!

Youth Focus - - Annchen Knodt

Annchen Knodt of Brenham is a senior at Brenham High School. She is the daughter of Tom and Diana Knodt and has a sister, Anya, 16. She has performed volunteer work at the hospital Wednesdays for the past two years. She is a member of Grace Lutheran Church, where she has played the handbells in the church choir for the past two years...

Q: What is your least favorite thing about high school? "Standardized tests."

Q: What is your biggest accomplishment? "When I aced the SAT test."

Interesting. She doesn't like the standardized tests in schools, yet her biggest accomplishment - and this from someone who plays handbells, served as drum major, and does useful volunteer work - was acing a standardized test. I don't understand. Haven't we heard all along from the educrats that children and teenagers suffer irreparably when asked to do things they don't like, or things that might be too challenging? Haven't we been told that self-esteem comes from being told you're perfect just as you are, rather than from things you do? And yet here we have someone who acknowledges that the high point of their life, so far, was successfully tackling something they didn't like very much at all, but which was important, and needed to be done.

Here we have someone who gets it.

Head Start results looking good

Testing of preschoolers for the Head Start program regularly comes under scrutiny; this Newsday article is the latest example. Thankfully, no tears are featured.

...many Head Start staffers on Long Island and elsewhere question whether children so young should take standardized tests at all. Scores now are rolling in from the U.S. government's first-ever tests of more than 400,000 preschoolers nationwide -- one of the biggest federal educational assessments in history. The new tests are the brainchild of the Bush administration, which seeks to measure whether the $6.8-billion Head Start program is doing its job in preparing impoverished youngsters for school. Nationwide, the 40-year-old antipoverty program enrolls more than 840,000 children, ages 3 to 5, including about 2,000 on the Island. So far, test results look mostly positive...

Established in 1965, Head Start started out focusing on children's socialization skills, such as following directions. The program gradually has grown more academic. Skeptics in the education field wonder if this particular test is the best way to measure performance, though test sponsors say it serves as a useful reality check in combination with other assessments.

That's $6.8 billion, with a "b." I'd say a "useful reality check" is necessary to make sure all that dough isn't going down the drain.

When tulips and test forms appear

The news today is all about the new SAT, which will be administered for the first time next Saturday, March 12. Many newspapers are carrying articles like this one (featured at Inside Bay Area) on the pros and cons of the SAT in admissions.

Spring's a big time in testing whether or not there's a new SAT. The FCATs start on Monday. The benchmark exams in Arkansas are next week. The Ohio Graduation Exam will be given on March 14, and will count for the class of 2007. I'm sure there are plenty more I'm not listing here.

One bit of news with an interesting twist - the University of North Dakota, and North Dakota State University, will not require the new essay portion of the ACT for admissions. Mainly, it's because they're interested more in developing their own essays - and not penalizing students who took the old ACT without the essay - but I found this comment from the student body representative to be amusing:

In November, the North Dakota State Board of Higher Education approved a new policy asking the state's 11 public universities to set the requirement. However, the board left the door open for schools to set exemptions. UND and NDSU's stop-gap approach was needed because the schools didn't have enough time to get formal, more permanent exemption policies in place. Many high school juniors have already taken the ACT without the essay...

Jordan Schuetzle, UND student body president, said he's pleased with the UND administration's response. UND's Student Senate opposed the essay requirement because of its cost and because it added a layer of subjectivity to a standardized test. Schuetzle added that the requirement is unfair because it gives an advantage to larger, more affluent school districts with means to hire quality writing teachers and coaches. He said also it might impact enrollment.

If I read this right, Jordan believes it's unfair for people who write better to have the edge in college admissions, all those remediation classes be damned. He seems to think that the ND administrators agree with him, but it sounds like they're planning their own much more substantial writing exam.

AIMing to keep AIMS

In Tucson the natives are restless:

Legislators are wiggling worse than this year's high school juniors over the status of AIMS as a graduation requirement. The class of 2006 will be first to face passage of Arizona's Instrument to Measure Standards as a prerequisite to graduation.

The kids get five tries. They already have taken the tests twice, and about half of them didn't pass all parts of AIMS. Then again, they took the tests as sophomores. They have until senior year to pass all parts. And they know that they must conquer AIMS to graduate.

Or do they? Some lawmakers insist the kids should get diplomas without passing AIMS. So instead of studying, many juniors may be crossing their fingers and biding their time.

Talk about mixed messages. Reminds us of the mom who tells her son he can't have the car till he finishes his homework, then tosses him the car keys. Please. Our kids deserve consistency - and not just at home...

The test series isn't perfect. No standardized test ever is. But we likely will spend eternity refining AIMS to better reflect evolving state standards. In the meantime, the least we can do is keep a strong and certain stance with our students. They know what to expect.

And our lawmakers have chosen precisely the wrong year to mess with those expectations.

What's gotten Tucson editorialists so steamed? My guess is, events like this:

Disabled and special-education students would be exempt from passing AIMS to get a high school diploma under a bill approved overwhelmingly yesterday by the House K-12 Education Committee. Students who meet other criteria would be exempt: those who had near-perfect attendance, completed all required courses with C grades or better, took AIMS every time it was offered and participated in senior-year AIMS tutoring...

The legislation is a watered-down version of a bill (HB2294) that would have retained AIMS, a reading, writing and math test, but eliminated the graduation requirement.

The editorialists aren't the only ones aggravated by this:

State Senate President Ken Bennett said he would prevent his colleagues from debating a proposal to let many high school students graduate without passing the AIMS test...

The lawmakers sponsoring the plan say the exam has taken away autonomy from local school boards and that they will continue pushing for their plan. They appear to have the support of about half of the Senate and a large majority of the House.

Bennett said he would halt any attempt to dismantle AIMS as a graduation requirement unless there are other legitimate alternatives in place, such as a specified score on college placement exams such as the Scholastic Aptitude Test...

Schools chief Tom Horne isn't kidding around, either:

The class of 2006 must pass the high school AIMS exam to graduate, and now Arizona schools chief Tom Horne wants each student's AIMS scores stamped on his or her diploma.

Critics say the proposal would unfairly rank kids, and students would be less likely to show off their diplomas to families and friends on graduation day. Horne says it would motivate students to study harder and reach their potential. He will ask the state Board of Education to consider the plan when it meets Monday...

Under the proposal, districts would be required to place a sticker on each diploma, showing a student "met the standards" or "exceeded the standards" on the math, reading and writing sections of the exit exam. For example, kids who "met the standards" on last year's math section scored 70 percent to 83 percent; those who "exceeded the standards" scored 84 percent or above. A graduate who exceeded standards on all three sections and made A's and B's in college prep courses would get a "high honors" sticker.

...and you can just imagine what types of responses this plan got:

Linda Ronnebaum's stepdaughter is a junior at Deer Valley's Mountain Ridge High School, and Ronnebaum fears the stickers would become a competition and a stigma. "Once you start putting labels on diplomas, you're labeling children," said Ronnebaum, an administrator at a West Valley elementary district. "Let's make them feel good about getting their diploma, not comparing what stamp you got on it."

Mary Silva and Louise Silva are not related, but both have children who passed AIMS and are juniors at Agua Fria High School in Avondale. They had very different reactions to adding the sticker to their children's diplomas. Mary Silva worried that such individual labels would make a child "feel better or less than someone else."

"I know there are some children who study and study and just pass," Mary Silva said. "How does that make them feel?"

Let's take that thought even further. What about those students who study and study but don't earn a diploma? Aren't we all making them feel bad by giving those who pass a diploma in the first place? Shouldn't we have a piece of paper to give the failers to put up on their walls - something along the lines of, "You're perfect, just as you are!"?

Seriously, people. We're not talking about children here. We're talking about young adults - some of whom might enter military service immediately after leaving high school - who, if they haven't already, will soon realize that the world out there is going to let them know just where they stand in comparison to everyone else.

The basics

Education author Bernard Chapin (who's been featured on here before) hits the nail on the head in a description of why touchy-feely educrats don't like standardized tests:

I had a chance to look at one of the Illinois books that involve “teaching to the test” the other day. It was quite informative. It contained exercises in phonological awareness that are intrinsic to any child’s learning how to read. I was rather impressed. The real thing that progressives hate about standardization is that it takes warm and fuzzy out of the process. All that is left is what you know and what you don’t know. Standardization merely compares the performance of an individual to that of a larger population. That’s it. Yet, comparisons of any kind are despised. It takes all the excuses and variables out. If a child cannot read as their peers do then is their manner of reading acceptable? No. Is it what we would call reading at all? No. That the testing of students is counter-productive is a widespread belief in our schools...

When I taught at the university...I had a student tell me that her brain did not think that way in regards to the multiple-choice examination that she got a C on. What processing deficit could she have had possibly had? All she had to do was select the right answer amid three wrong ones. It was child’s play, at least many years ago it was child’s play, now children, and some adults, are not expected to do it.

Emphasis mine. Standardized tests cut through a lot of the sheep dip in "progressive" educational techniques.

The stylish way to protest the test

Hoo-boy.

Maeghan Gibson is fed up with the state's standardized test. Focus on the Texas Assessment of Knowledge and Skills is encroaching on classroom learning, Gibson said. Instead of having high-level discussions, the Haltom High School junior honors student said she spends too much time taking practice tests and filling out work sheets.

So she and a few friends waged a silent protest Monday morning outside the school by handing out pre-sold green T-shirts with slogans including "Walking standardized test score," "I am not in the equation of my education" and "Total Annihilation of Knowledge and Skills."

I want one of those "Walking standardized test score" shirts. And, frankly, students should be free to wear them - even during test time.

The students say their protest was not aimed at Haltom High School, teachers or the Birdville district, but rather at state and national policies that require the standardized test.

"It's turned into a real 'Grab a work sheet, go sit down and you have to know this or you will fail' kind of thing. That's not good for long-term learning, in my opinion," junior Chase Robinson said. "We want our teachers to advance our knowledge, not a test."

And that's commendable. But long-term learning doesn't often happen in the absence of basic skills, which is what these types of tests are intended to measure. It's not surprising that some of the more bored students are fed up with all the tests, but it would be pretty tough to, as one student suggests, create a test that measures only whether a student improves, and not whether they fail.

Bubble sheets and bunny feet

Poptarts and pajamas help kids prepare for standardized tests:

At Sharpsburg Elementary, girls and boys giggled at each other's pajamas and slippers on the way to class Saturday morning. A couple of boys lugged boxes of Pop Tarts and fruit drinks to three classrooms, as about 30 fourth-graders settled into chairs or on sleeping bags and played "Survivor" and "Who Wants to be a Millionaire."

It seems to be working. After three years of Saturday pajama parties, Sharpsburg went from not meeting statewide standards to meeting them all and earning an "excellent" designation last year, said Principal Brad Winterod.

Next up, "Truth or Dare" with math questions.

Criticizing the tests the correct way

Devoted Reader Daryl sends along this article about the new SAT:

I've been a high school English teacher for 10 years, and if there's one thing I hate worse than the SAT, it's the idea of a new SAT...It's not that I'm against assessing kids. I give my own students eight to 10 assessments each marking period, though my assignments don't look anything like what students encounter on these high-stakes national exams...

What the bulk of the writing section of the new SAT is really measuring is acquired skills in managing style within the realm of standard written English...Students would be better served by consistently reading the commentary section of the local newspaper - and then periodically writing letters to the editor - than by sitting through the painfully boring lesson plans that these changes to the SAT are likely to inspire.

I agree. And all those testing critics out there should be taking notes - THIS is how effective test criticism should be done. No hyperbole. No hysterical rants about how the exams are completely biased or dependent on one's social standing. No childish arguments about how evil it is to hold students to objective standards.

It's completely legitimate to worry about the impact of the new SAT on writing curricula. It makes sense to put the horse before the cart and suggest changes that schools can make to improve writing skills. And portfolios, though expensive and laden with plenty of psychometric challenges, are not a bad idea for writing assessments.

Public vs. private gaps

We've all heard about the test score gap (aka, the achievement gap), and Andrew Coulson of the Mackinac Center has a provocative article about how ideology helps the gap persist:

As researchers know all too well, there is still a gulf of more than 220 points between the SAT scores of white and black students, and black children trail their white peers by significant margins on every subject tested by the National Assessment of Educational Progress. Many people are likewise aware that Michigan performs even worse in this regard. Across grades and subjects, Michigan’s racial achievement gap on the NAEP is four to nine points larger than the gap nationwide.[1]

But there is one aspect of the achievement gap that is almost universally unknown: how it differs between public and private schools.

Coulson uses NAEP data to reach a conclusion that isn't really surprising:

As the table shows, there is a sizeable achievement gap between black and white fourth-graders in both public and private schools. It is also clear that the private-sector racial achievement gap is narrower at the 12th grade than at the 4th grade in all of the core NAEP subjects. Public schools actually see a larger race gap in both writing and mathematics at the 12th grade than at the fourth.

Note that the achievement gap does not close faster in private schools because white private school students lose ground with respect to white public school students as they move to higher grades. Rather, the gap closes because black private school students have learned at a substantially higher rate than black public school students.

I like his concluding remarks:

So, will the NAACP and other groups avowedly committed to reducing the racial achievement gap act on these findings? Will they compete with one another to discover the best way of bringing nongovernment schooling within reach of all children?

Given the choice between actually narrowing the racial achievement gap and remaining ideologically pure, they will chose ideological purity. Sooner or later, this position must surely crumble under the weight of its own immorality.

Emphasis his. He also notes that the dropout problems with public schools probably give a boost to the public school scores, because the worst 12th-graders aren't there to take NAEP. Yet, the private schools look better.

As for his belief that this position will crumble soon, let's just say I'm not holding my breath for that, any more than I'm holding my breath for the eradication of the racist belief that minority children cannot be expected to do well on standardized tests.

First homeschooling, now home prepping

Test prep reaches the little ones:

Steve Gold and his son Austin studied an hour every school night for weeks to prepare for the big test. It wasn't a final exam or an SAT.

Austin is a fourth grader in Rockland County. They studied for the New York state assessment test on English Language Arts given this week.

Welcome to test prep - grade school style.

You can tell this article isn't appearing in the New York Times; if Winerip were reporting this, there would have been tears aplenty by this point in the article. The NY Newsday, however, sticks to the facts:

As standardized tests proliferate at the lower grades, study guides and programs have filled a growing market...The market is expected to expand when standardized tests for grade schoolers become the rule this fall under federal No Child Left Behind legislation...

So is test prep at home even needed?

Good question. I say yes. The parents need reassurance, and the kids need practice. Good test prep books will be able to fill both sets of needs. The critics, of course, say that the test prep must inevitably be coming at the expense of "education," as though learning to read and learning how to take a reading test cannot possibly cover the same constructs.

Some parents won't buy the books, because they'll see their kids don't need them. But these books could be a boon to a parent who is unfamiliar with the tests and/or believes their child needs extra help.

When a math exam becomes an exercise in basic probability

The New York Post has the goods on the new but not improved Math A Regents exam standard :

High-school students taking the Math A Regents exam this week must correctly answer fewer than one-third of the questions to pass — the lowest benchmark in at least six years — because of a revised grading scale that critics charge is too generous. Students are required to earn just 26 out of a total 84 points — or 31 percent — to reach the minimum passing grade of 55.

The state Department of Education insists the scoring is just as rigorous as previous exams and consistent with recommendations of the independent panel that devised the test. "There are somewhat more difficult questions on this exam, however, and so students need to get somewhat fewer questions right in order to pass," said state education spokesman Tom Dunn. "Anyone who looks closely at this exam will see that it is not easy."

I disagree (as does the Post), and for one simple reason. We heard a similar argument a while back, in the discussion of the UK maths GCSEs. If you recall, I said then that there might indeed be no problem with a very low passing standard, whereas here, I say there is.

The difference is that, while the UK exam is completely open-ended, the Math A Regents exam contains thirty multiple-choice items (or MCQ's, as we call them) that are worth a total of 60 out of the 84 total points - 71.4% of the exam. If the passing standard is set at a raw score of 26 points, or 13 MCQ's answered correctly. A student can pass this exam by getting 13 MCQ's and no open-ended items right, because all items are combined and scored; the conversion is done only on that final combined number-right score.

Mere guessing, at four options an item, gets a student up to 7.5 items by chance alone. It doesn't matter if the MCQ's are more difficult; no matter how difficult they are, students can still get the right answer by chance alone. So a student need get only an additional 6 items right in order to pass. They can completely bomb the open-ended part and pass. And calculators are allowed. And this represents an increase in the number of MCQ's on the exam compared to 2003. And we haven't even gotten into discussing the generous conversion table.

I'm sure you can guess the reason why the passing standard was lowered, by the way. They've been very concerned in NY over the miserable passing rates on this exam. This solution doesn't exactly address the root causes.

Others who have more math knowledge and understanding of these exams than I have been appalled by Regents math exams in the past. My guess is that they're still appalled. If you haven't visited the NYC Hold website, you should. It's a great read.

(Via Joanne Jacobs.)

Meet the only man on earth willing to publicly defend SAT analogy questions

A test-prep author makes the news:

...Because of the changes [in the SAT], previously used methods to taking the exam were cast aside, opening the door for Christopher Black, an independent education author in Greenwich, who has since revolutionized the way students approach the SATs.

"Of course they were wrong," said Black, who used to substitute teach at Greenwich High School and Central Park East before receiving his master's in education from Columbia University. "But their approach sold a lot of books and prep tests"...

Yes, it certainly did. But Black's approach, so far, seems to be showing results, not just profits. Not surprising, considering that it focuses not on tricks and insider tips, but on "reinforc[ing] the fundamental academic reasoning skills" used on the exam. In other words, by actually teaching would-be test takers something useful.

After the perfect score, what's next?

The bar keeps getting set lower as the kids who make perfect SAT scores keep getting younger:

A 13-year-old boy has scored a perfect 1600 on the SAT, a standardized test usually given to college-bound high school seniors.

In June 2003 his brother Ross scored 1600 on the SAT as a junior at Mechanicsburg High School. The oldest brother, Alan, had 1520 on the exam.

Oh, I get it. This kid was just NOT going to let his older siblings show him up. Good for him. And why wasn't he chosen to play Harry Potter? (He already owns a magic wand - his pencil.)

On a related note, I wonder if anyone (other than ETS/College Board) has kept track of the number of students who achieve perfect scores each year? And what would it mean if that number has, for example, drastically increased as of late? The result of "dumbing down the test?" The result of those who might do poorly being more likely to bypass the exam for schools that no longer require it? Or the natural by-product of an across-the-board increased focus on standardized testing?

Quibbling over the data

FairTest has been posting SAT score tables on its website which show that poor and minority students do worse on the exam than white, wealthier students, and the College Board isn't happy about it. It's a matter of debate as to whether this is an indictment of the tests (as FairTest maintains) or of our educational system (as I maintain), but, interestingly, that's not really the point of the latest controversy:

The nonprofit College Board, which owns the SAT college entrance exam, has demanded that its chief critic remove from its Web site data showing that minority and poor students scored lower than white and upper-class kids. The letter to the National Center for Fair and Open Testing, also called FairTest, claims the Cambridge-based nonprofit organization violated copyright law by posting the scores without permission.

FairTest, which opposes an overreliance on standardized tests, posted the Oct. 27 letter on its Web site along with its refusal to comply with the College Board's demand. FairTest argues that the data is widely available in the public domain and therefore not subject to copyright protection.

Needless to say, this gives FairTest the opportunity to get up on their soapbox about how CB don't want any negative information about their test to be released, but the public is already quite aware of test score gaps.

Critics long have attacked the tests as unfair, chiefly because white students tend to do better than other groups. Many reasons are offered -- family income and education, school quality, courses taken, access to tutors and test-prep courses.

If they've "long attacked the test as unfair," it's hard to believe that CB is really trying to "hide" something from us now. This sounds like one for the lawyers to sort out. It's interesting, though, that the AP doesn't see fit to point out that three of the six "unfair" reasons above - courses taken, access to tutors and test-prep courses - are directly related to the amount of effort a student is willing to put into their education. Saying it's "unfair" for a student who takes more difficult courses, spends extra hours with tutors, and prepares for the SAT, to receive a higher score is ludicrous.

If anything, the list of reasons above suggests a great deal of validity evidence for inferences made with the SAT, because we'd expect those with more money, better schools, harder courses, and more preparation to do better. It's not the score gap that is seen as unfair so much as the disparate opportunities that lead to it.

Teaching to the (math) test can be a good thing

Mobile Register reporter Rena Havner tries hard to make the scenario of "teaching to the test" seem disastrous, but it honestly doesn't seem to me like the kids are being disadvantaged by an increased focus on long division:

Two years ago, before the Mobile County school system began administering standardized quarterly tests, third-graders weren't taught long division until the last two weeks of school, [teacher Cindy] Naylor said. Now, with the federal No Child Left Behind Act and a greater emphasis on standardized testing, long division must be taught months earlier, before the children are mentally ready, Naylor said.

Naylor, who has taught for 24 years, has told the Mobile County school board that too much emphasis is being placed on standardized tests and that children, and their grades, are suffering...She said she now has to "teach to the test," meaning that her lesson plans revolve around the subject matter covered on standardized exams that the system uses to evaluate teachers and that the state uses to rate schools.

Are we supposed to assume that a test for third-graders is bad if it requires they learn long division? I think so, but it sounds to me like many of Naylor's students are benefiting from this new structure:

She undertook plenty of tactics to try to make the long math lesson sink in: To prepare the children for long division, she read a story about ants that must find a number by which 25 can be evenly divided before they can march in a parade.

"Seventy-two!" the boy said, bragging to his classmates. "That was a hard one, but I got it right."

Remember, this is a kid who wouldn't have been taught long division until sometime next year. And now he's thrilled about the fact that he can do multiplication in his head, on command. I realize Ms. Naylor does have a lot on her plate, and I'm sure the paperwork associated with NCLB can be overwhelming. But I don't get the feeling that her students are suffering because of it. Ms. Naylor feels she's not teaching as much "constructive" information any more, but the lesson plans above suggest she's doing a bang-up job in getting her students to understand the rules of mathematics. It's hard to see where the problem is with that.

Graduate school: self-discipline required

Howard University's newspaper, The Hilltop, has an article about the "unfairly" high prices of test prep courses:

With many juniors and seniors preparing to take the Graduate Record Exam, there are many complaints about the high cost of preparatory classes. The GRE is a standardized test that provides graduate schools with a way to compare qualified applicants for admission and financial aid. Many students take these prep classes to help them get an idea of the format of the tests, to get practice and to help them maximize their scores. The prices of these classes can range from $400-$2,200 for 15- to 35-week courses.

These prices are out of the budget range of many college students who have to work and save or depend on family members to give them the money. This may force many students to opt to study for the exam on their own instead of forking out the several hundred dollars for a structured class.

Is it just me, or does that sound like we're supposed to believe that being "forced" to study for the GRE on one's own is a bad thing? It seems to me there is at least one erroneous assumption at work here, which is that there are special, hidden tricks to these standardized tests that only the test prep coursemakers know. What's more, these tricks must be very valuable, because Princeton Review is charging so much to reveal them.

I believe nothing could be further from the truth. The reason PR and their colleagues get away with charging this much is because students are willing to believe the anti-testing hysteria that's shoved down their throats daily. If you were convinced that psychometricians specifially write test items that female or minority students can't understand, you'd be rushing to sign up for test prep too, and hang the cost.

What's more, these aren't children we're talking about here, but young adults who are preparing for graduate school. Graduate school tends to require a great deal of self-discipline, logical thinking, concentration, and motivation. Although programs do what they can to support their students, when you come right down to it, the only person who can really shape your path towards a Ph.D is you. Anyone who can't get through the GRE without a lot of expensive outside help might find themselves floundering in the sink-or-swim world of a graduate program.

When you need bifocals to take the exams

Hopeful college applicants aren't the only ones who should be prepared when it comes to high-stakes tests. This Globe and Mail reply to a letter-writer concerned about employment personality testing is quite interesting:

Strictly speaking, you can refuse or withdraw from testing at any time; no psychologist worth her salt can make you sit through an assessment against your will unless you're ordered to by the court. But refusing to be assessed in a job competition is declining to play the game by the employer's rules...

If asked to recall how many statistics courses were required before they were handed a diploma, most psychologists are likely to respond with eye-rolling, grimacing, tongue protrusion and other signs of disgust.

Well, they got THAT part right. I've known too many school and clinical psychologists who hated statistics, and avoided statistics classes like the plague. They'd take the bare minimum and gripe about it the whole time. Now, I don't expect everyone to be as interested in psychometrics as I am - but still, I find troubling the anti-stats mindset that you see in abundance in psychology undergrads, and to a lesser extent in graduate students.

Not all employment testing is hogwash.

Intelligence tests are good predictors of job performance precisely because it's your level of intelligence that allows you to acquire job knowledge and skills.

Problem is, it's un-PC to say that nowadays.

"Eke"-of-grade testing

In the past, some less-than-motivated students have been able to use a loophole, related to North Carolina's extensive end-of-grade and end-of-course testing, to avoid working consistently throughout the year. Students in Orange County were able to squeak by and pass classes with high test scores - but low homework scores - and now administrators are looking to close that loophole:

Occasionally, an Orange County student who's hasn't done his or her classwork during the school year has wound up passing the grade or class by acing a state-required standardized test.

"You potentially had students who would do nothing during the school year," said Superintendent Shirley Carraway. "It's just awful. If you think the end-of-grade or end-of-course test is a minimum standard, you could pass and simply eke by."

Board members are now aiming to make promotion dependent on any local requirements as well as test scores.

Helping NYC's third-graders

Newsday wonders whether the high-stakes exams for NYC's third-graders are working:

Nearing the end of a 45-minute intervention session in PS 100 in Queens, Amy Strauss looked into the faces of six third-grade and fourth-grade boys...The children, using a dozen block letters, had been asked to form three-letter words on their white magnetic journals.

"'Pat.' The next word is 'pat,'" the teacher said slowly. "P ... A ... T." Choosing from a selection of nine consonants and three vowels, five of the boys spelled the word correctly. The sixth boy spelled "p-i-t."

The boy who had misspelled the word, mumbling to himself and following Strauss' example by tapping his own fingers, removed the "i" and inserted the "a" block in its place. "Very good, everyone," Strauss told the group approvingly.

Remember, these are third-graders, who aren't being asked to write out letters, or even know how words are spelled without being told. They're being asked to identify letters of the alphabet from a reduced set of block letters. Is there really any question about whether a kid who has trouble with this should be promoted to fourth grade?

The intervention sessions, held during the school day as part of the students' regular curriculum, are another crucial part of Mayor Michael Bloomberg's and Schools Chancellor Joel Klein's efforts to end so-called "social promotion," to keep failing students from being passed to higher grades.

It's not "so-called" social promotion. When failing students are promoted, it is social promotion.

Citywide, more than 3,600 third-graders were held back -- or "retained," in the Department of Education's words -- after posting Level 1 scores on a city standardized test in reading and math. Level 1 is the lowest of four levels on the test. Students who posted the low scores on the test last spring were required to attend six weeks of summer school and make a higher grade upon retaking the test to go on to fourth grade. In a small number of cases, students who had low scores successfully appealed and were allowed to advance...

Many education experts, however, say the jury remains out on the intervention programs until the citywide standardized exams are given to third-graders in the spring...Some third-grade teachers have had mixed feelings about the intervention programs, saying whatever improvement a student who is getting extra help shows is countered by the chaos of being moved around too much during school hours. One teacher, who asked not to be identified, said he found the interventions disruptive because, among other reasons, the push-in programs took away from his lesson.

I agree that the jury is still out until the test scores come in - even if the idea of intervention sounds good to us, that doesn't mean that what's being taught is effective. However, I think it's absurd to say that, if the intervention work is useful, it's negated by being moved around a great deal. And if the push-in programs take time away from lessons that aren't working, as evidenced by test scores, I don't see where that's a bad thing. Teachers should be kept more informed about how the intervention programs work, for sure, but if classroom lessons aren't working, there's no reason not to try something else.

Twins take ETS by storm

They're cute. They're hip. They're awfully smart. And they both did pretty darn well on the SAT:

It seems like the kind of SAT question custom-made for Dillon and Jesse Smith of Long Beach: If one out of every 1,511 students taking the SAT will get a perfect score, what are the odds that twin brothers will both ace the test? nswer: No one knows for sure. Nevertheless, that's what the Smith twins have done.

Both Dillon and Jesse Smith, 16-year-old fraternal twins, achieved the elusive top score of 1600, a number most high school seniors dream about seeing on their SAT score report. "I was very, very happy," said Dillon, describing the moment he realized that both he and his brother received the top score on the aptitude test. "I've been hoping for it since we started."

It was a rare thing to hope for. Of the 1.4 million high school seniors who took the test in 2004, only 939 scored a 1600, according to the College Board, which administers the test. With those numbers, the odds of any two people getting that score would be almost 1 in 2.3 million -- and that doesn't even take into account whether those two people are related, never mind twins.

No, it doesn't (although, as my Devoted Reader Maureen points out, the Newsday reporter gets it wrong by implying that the probability would decrease if family relations were factored in). Dillon and Jesse don't have identical DNA, but they did share a tough mom with high expectations:

"I expected it," said Smith, 44, a physical education teacher at I.S. 143 in Washington Heights. "They have the potential to do even better -- maybe even write the tests ."

What? Momma Smith thinks they could become psychometricians one day? Be still, my beating heart. Oh sure, Jesse is quoted as saying that he doesn't think the test really measures anything, but I figure he's just being modest and humble. After all, I've yet to see the article where a perfect-scoring student is quoted as saying, "Yeah, this test really measures how smart you are, and this proves I'm the best!" I know some top scorers think that, but they'd never say it.

I always wanted to wear a tiara and sash

I'd always thought it would be a tad surreal to win an Academy Award, or an Olympic gold medal, or some such spectacular trophy for all-around splendiferousness. Surreal because of the rush of adrenaline that comes from all the admiration, knowing you are appreciated - but also knowing that you better get off your duff and do something else so that you don't seem like a flash in the pan.

It's even more surreal, I can now say, to receive a lovely award for which you didn't know you were being considered, and in fact didn't know existed. I fired up my computer this morning to read the following email from Daryl Cobranchi:

Kimberly,

Daryl

Wha? What? Through my blurry and uncaffeinated eyes I stared at the screen. I couldn't figure out what Daryl was talking about. There was an email immediately preceding his with the subject line, "WINNING NOTIFICATION!!!", but even an uncaffeinated hedgehog could've figured out that that was one of those Nigerian-lottery scams. Being not quite as sharp as a hedgehog this morning, I clicked on it, just to be sure:

ATTN;WINNER, We happily announce to you the draws of the Centenary Olympics Big Lottery International programs held on the 25th of OCTOBER 2004 here in Athens, Greece.This promotion takes place every Olympic year You were entered as dependent clients with: Reference Number:NM/BC921245/KY13, and Batch number NM/207161/KOP....

Okay, so it's not that. I went to Daryl's site, and discovered that Education News has declared N2P to be "The BLOG of The Century." They don't say which century; I hope they don't mean the last one:

In Defense of Testing Series
Number 2 Pencil: The BLOG of the Century
Tuesday, October 26, 2004

It seems hard to believe that Kimberly Swygert's BLOG, Number 2 Pencil, approaches its third anniversary. It still seems so new. For those of you unfamiliar, Kimberly is a card-carrying psychometrician who expresses in her BLOG her open, honest, and informed opinions on education policy in general and standardized testing in particular. Before she started, no psychometrician had been willing to do this, in the interest of protecting their careers. We all owe her a debt of gratitude for her courage.

Wow. How flattering. Especially given all of the other edublogs out there (see the list over on the right-hand side of this page if you don't know about them). And they got the timeline right, too - N2P will indeed celebrate its third anniversary in January of 2005. It sounds amazing even to me, especially considering that I thought no one would ever read this blog. Well, I thought maybe some other psychometricians might read it, but no one else.

What a nice way to start the day. A bit surreal, as I said, but nice. Had I known I was in the running for such praise, I would have tried a bit harder and posted a few more testing-related articles, at the expense of a few catblogging posts. But hey! - I made the Carnival of the Cats this week too. So I must be doing well all around.

Now would be a good time ("I'd like to thank the Academy...") to thank those who have provided a great deal of academic, moral, and technical support for N2P, almost since day 1: Joanne Jacobs, Dr. Greg Cizek, Dr. Richard Phelps, John Rosenberg, Dean Esmay, Daryl Cobranchi, and every single one of my Devoted Readers. Without you, this blog would be nothing.

This post will stay at the top of the page for the day; scroll down for new posts.

After the big waves come the waivers

Thanks to the recent stampede of hurricanes through Florida, some schools may be able to obtain a "storm waiver":

State education officials said Tuesday that some hurricane-ravaged schools can appeal their grades next summer if students show unexpected declines on the Florida Comprehensive Assessment Test. But the same consideration doesn't extend to the students themselves.

To be eligible for a grade waiver, schools must have been shut down more than five days by a hurricane, have shown good or improved grades over the last three years, and drop at least a letter grade next year. They must also show obvious effects of hurricane damage, such as a high number of dislocated students or classes on double sessions.

One other solution: Change the FCAT to measure how well students know to board up houses, purify drinking water, and plot evacuation routes. You know, the skills children need to best function in Florida.

Skipping out on the exams

Fellow blogger Liz Ditz emailed me to point out that students are skipping out on exams that are low-stakes to them, but high-stakes to the schools:

Torrey Pines High School [CA], the academic powerhouse proclaimed by Newsweek as among the top 100 campuses in the country, is in the academic doghouse. Nearly 300 students at Torrey Pines skipped the statewide standardized tests in the spring, and because of low turnout, one the highest-achieving high schools in California didn't receive a statewide ranking, known as the Academic Performance Index.

Principal Rick Schmitt said the API score is a symbol of a school's academic standing. It affects property values and is used by real estate agents to sell homes..."Individuals feel there's nothing in it for them," said Schmitt of students who are expected to take the tests. "If the community is better informed and once people understand what it means on a bigger scale, they can appreciate why it's important."

Well, yes, but I can see why students skip. Trying to get good data without stakes for the test-taker has always been a thorny issue, and there's not any one solution that works for all tests. When students get fed up with tests to the point of drawing cool patterns on their bubble sheets, something's gotta change. The data are worthless at that point. Students must have an incentive.

They study hard for the SAT and sweat over the APs; they at least show up for the high school exit exam. But yet another test that doesn't matter at all, to them, might be too much.

Stressing out in Virginia

Devoted Reader nicksmama sent this along with the headline, "Bad test-takers rejoice!":

Diane Smart has seen the stress and strain Praxis I puts on some teachers. For their sake, she's hoping the Virginia Board of Education lowers the scoring standards later this month for the general-knowledge teacher assessment test.

Virginia has the highest minimum required scores of the 28 states that use Praxis I. While most teachers pass the state requirements, others struggle.

And...isn't that to be expected? Otherwise, why give the test? Oh wait, I get it - everyone who has a love of teaching should be allowed to be a teacher, right? No matter how much - or little - material they've mastered.

On Oct. 28, the state Board of Education will consider lowering the standards in one or more of the three assessment areas...The standardized test is similar to the SAT. Each section takes about an hour to complete.

The average requirements in other states are about 172 for math, 174 for reading and 172 for writing.

Emphasis mine. Care to tell me why men and women with bachelor's degrees shouldn't be expected to do well on a test of ninth-grade math?

Smart thinks the discrepancy is unfair and can force people to take the $130 test multiple times. She points to No Child Left Behind, which is the same in all states, and says Praxis I requirements also should be uniform.

"Unfair"? Really? Why? Because someone always fails it? And I thought teachers disliked NCLB because it imposed government standards on student performance. Now we hear that teachers want government standards on teacher performance?

Despite Virginia's allegedly-unreasonable high standards, it's not like massive numbers of would-be teachers are failing it. But the statistics are telling:

According to 2002-03 statewide data, about 92 percent passed the reading section, and 86 percent scored at least the minimum on math. People fared worst in writing, with 82 percent passing.

If you want to teach children, you should be required to pass any and all writing exams the state puts before you. End of story. Writing well is not an elective skill for a teacher of any subject, at any level. The ACT and SAT are alternate exams - and they both have writing components.

Lending a helping hand

Cal State would like to reduce the numbers of incoming freshmen who require remedial classes, so they're doing early assessment on California juniors. The test used is an extended version of the state standardized exam, and the results are depressing:

Just more than half of California 11th- graders who took an early assessment test were college-ready in math, and only one- fifth were prepared in English, according to results from the California State University's new high school testing program.

Forty percent of California high school juniors this spring volunteered to take part in CSU's early assessment program. Fifty-five percent of math test-takers were deemed college-ready, as were 22 percent of students tested in English.

So these were the kids who were motivated and organized enough to volunteer. Presumably all of them are planning on attending college. And yet only 22% are ready in English? Good heavens. Cal State hoped to identify the problem areas, and they certainly have. What they do with this information now will be the crucial part.

A partnership between the Long Beach-based state university system and California's K-12 schools, the program aims to smooth the transition from high school to college by giving struggling eleventh-graders extra help during their senior year so that they can avoid remedial course work if they are admitted to a CSU...

To help struggling students during their senior year, CSU faculty have developed a new 12th-grade writing and reading course and have set up Web sites offering math and writing tutorials and feedback.

The diagnostic and tutorial services for reading and math that CSU is providing look to be substantial, and solid. I believe, though, that this project is essentially a tacit admission by CSU that college-bound students are no longer expected to acquire the necessary academic skills in high school. Some students have always needed extra help to reach the college level, and certainly high schools do not design every curriculum as college-prep worthy. But a program on this scale is obviously intended to remediate a large number of students who will earn a California high school diploma and expect to attend college, yet have not been adequately prepared.

Is it a better idea than just offering remedial classes in college? Yes. Should the high schools involved feel just a tiny bit of embarassment that this is so necessary? Yes to that, too.

A budding psychometrician

Holy cow. It's the first-ever, "I want to be a psychometrician" article that I've ever seen:

Sara Monempour was 2 when her family moved from Tehran to L.A. Then she did what most new Americans do: learned English. Attending Los Angeles County public schools, Monempour excelled in class but scored "unbelievably low" on standardized reading tests, up to and including the SAT.

Now 23 and a doctoral student at the University of California-Los Angeles, she hopes to become part of a small but growing group of elite researchers, known as psychometricians, who do little else but think about standardized tests.

"Do little else." I love it. I guess that's supposed to seem bizarre to the average reader...well, I guess it is bizarre.

No Child Left Behind, President Bush's education reform law, more than doubled the number of standardized tests schools must give each year, and it very likely will double it again in coming years.

Only about a dozen universities have psychometrics programs. Most turn out only one or two graduates a year. But once they're on the job, psychometricians burn out quickly, observers say. For one thing, new federal requirements say tests given in the spring must be processed before students return in the fall, months earlier than in the past.

Several small quibbles, already:

1. It's an article about psychometricians and the first quote is not from a psychometrician, but from someone who is rabidly anti-testing? (Never mind that, in this case, his quote is correct.)

2. "Burn-out quickly?" Funny, but I associate burn-out with police officers and neonatal intensive care nurses. Every single psychometrician I know is employed as fully as they want to be, and we all work long hours for organizations that are often under-staffed. But I can't think of one psychometrician I know - and I know dozens - who would consider him- or herself to be "burned out."

The rest of the article is fairly informative, including the following:

It's worth pointing out that even psychometricians say such tests shouldn't be the sole criterion determining whether teachers can teach, whether schools pass muster, students graduate or colleges accept them.

"Test scores are limited in what they tell you about a person, and test users don't always keep that in mind," Walker says.

True. Those tossing about overheated statements like "reducing a person to a number" are always testing critics, and usually uninformed ones at that. No psychometricians claim that everything about a person can be measured, nor do any claim that all decisions should be made quantitatively. If your college does well with students who bomb the SAT but can put together a great artistic portfolio, so be it. Don't use the test. Use what works. But don't claim the test is flawed for everyone just because it doesn't work with your population.

On that note, back to Ms. Monempour. I'm thrilled that she has noticed an issue with the tests. I'm thrilled as heck that she even knows what a psychometrician is, much less is actively studying to be one. But I do have one question. Given that she's noticed a pattern with all of her bilingual classmates, is she willing to enter the field of psychometrics with an open mind, and consider the possibility that schools these days may not do a very good job of teaching bilingual students Standard English? Or that some other factor may be at work?

The problems can lie with tests, with the education, or some mixture of the two. I have the feeling that the more involved Ms. Monempour becomes with psychometrics - and I wish her all the luck in that regard - the more she will notice that tests sometimes uncover ugly truths that others do not wish revealed. If she decides the problems are with the tests, I hope she sticks to her guns, and actively researches ways to fix them.

Show me some writing standards!

Colleges don't need a new standardized writing exam, claims the Southeastern Missourian:

...is there a need to better test college students' writing skills?

Southeast admissions director Debbie Below said the Cape Girardeau school already has its own writing assessment program and Southeast won't make prospective students spend additional money to take another ACT exam.

It appears that schools are not so much relaxed about writing as they are more anxious about other subjects:

Despite the new emphasis on writing, a spokesman for the Iowa City, Iowa-based ACT says skill with words is not the biggest failing of incoming college freshmen.

In contrast, only 40 percent of them are academically prepared to earn a C or higher in algebra, and only 26 percent are prepared to earn a C or higher in college biology, Gullette said.

Emphasis mine. According the Missouri Department of Secondary Education home page, the MAP requires Missouri students to know Algebra. I think. The homepage is confusing, and the math framework page is even more so. Page 59 finally gets around to mentioning some concrete examples, but it's hard to tell just how much algebra is tested at the high school level.

Anyway, some Missouri colleges say they have their own writing assessments, but the matter-of-fact description of why just about curls my hair:

Bratton said some colleges like Southeast [Missouri State] require new students to take a writing test to determine what freshman English course students must take. Others determine course placement on the basis of the ACT score on the English part of the exam...

"There are always going to be people who don't write very well," he said.

Funny, but I don't remember there always being college students who could not write in complete sentences.

Doing it right in Yonkers

One very disadvantaged school district in Yonkers (NY) is doing well on the state standardized exams, and enquiring minds would like to know why:

Seventy-seven percent of Yonkers elementary school students are now meeting the state's standards. That's compared to 79 percent of all elementary schools statewide -- including other large urban districts such as Buffalo, Rochester and Syracuse.

And results released in June show fourth-grade English test scores climbed from less than 34 percent of students meeting the state standards in 1999 to 65 percent this year.

From the press release:

A sample of the most improved schools, when contacted by the State Education Department, gave these reasons for improvement: an all-out district or schoolwide effort to improve achievement, hiring of math instruction specialists, intensive staff development focusing on math instruction, an improved curriculum in line with the standards, teaching math more every day, setting targets for improvement, using the achievement data to help individual students, grouping students flexibly by achievement level to give them the help they need, and before- and after-school help.

Emphasis mine. Hoorah.

Battling over the FCAT (again)

Hurricane FCAT is wreaking havoc in Florida's Pinellas school district - the nation's 22nd-largest school district with over 113,000 students:

In a major shift this year in Pinellas schools, teachers have been told to move much faster through lessons and to narrow their instruction to material most likely to be on the state's standardized test.

School Board members also are complaining, upset that administrators did not brief them on such a large initiative until they rolled it out across the district.

Parents, too, are only now being informed. But the district administrators say they had no choice - almost half of Pinellas students are not performing at grade level. (No, this is not how is has to be - the FCAT grade levels are not norm-referenced, nor set at the average or the median, so there's no reason for half the population to always be reading below grade level. In this case, standards are set such that every kid, theoretically, could perform at grade level or above. There are norm-referenced portions to the FCAT, but the Sunshine State Standards are benchmarks.)

The district's mantra when presenting the new program has been: "If not this, then what? If not now, then when? If not you, then who? Our students can't wait"...

In math, the same class needs to master 45-, 90- and 180-degree angles before studying other angles that likely won't be on the test.

And so the passionate statements flow back and forth; one side insists that this is "teaching to the test" and the other side says that this is the best way to get all Florida's students up-to-speed on the standards.

Putting pressure on the schools

In New Mexico, a school principal is fretting about the fact that his students didn't make the AYP (adequate yearly progress) targets this year:

Gonzales Elementary Principal Michael Lee knows exactly why his school didn't make "adequate yearly progress" this year under the federal No Child Left Behind Act: The fourth-graders didn't learn how to use rulers and measure quantities.

It is particularly painful to him, because his daughter was among the fourth-graders who took the test used to rate the school, and he thinks they're an exceptionally bright bunch of kids.

Dude. Fourth-grade is not too early to learn to use a ruler. It pains me to see these educators who insist that their students are smart - but just haven't gotten around to learning some basic facts and skills. If they really are that smart, heck, they should be converting inches to centimeters by now.

Still, Lee knows what his school has to do: Make sure this year's fourth-graders learn how to measure. Gonzales began a Cooking with Kids program this year, so the teaspoons and measuring cups they'll use in that program should help.

Well, that's something, but surely they could learn about measuring devices in math class as well, right? The article in general is good, but there are a few testing criticisms thrown in that, well, don't add up:

Sewing also sees problems with regard to students whose first language isn't English. Under the law, students have to start taking the standardized test in English three years after they enter the United States. Sewing said research shows language development takes five to seven years. "It's frustrating for these kids," she said. "It makes them feel like a failure."

"I would ask the adults out there, 'If you moved to China and lived there for three years, would it be fair to measure your education and your skills in Chinese?' " Sewing added.

If the entire time that I was in China, I was enrolled in a program that, for eight hours a day, was supposed to immerse me in Chinese and teach me the Chinese language, then yes, it would be fair. I might need a different standard than a native speaker - I can see the argument for that - but it's not unfair to test me to see if I'm where I should be after three years.

Gonzales Elementary's Lee said he'd rather "be sucker-punched" than repeat the experience of hearing his school had failed to meet AYP. "What we have to do at Gonzales is make darn sure we're teaching the standards. And because this test is based on standards, that should be easy."

"The pressure is going to build. And if principals and teachers are feeling pressure, then you can be sure that kids are feeling pressure. I have to wonder if this is the kind of world we want to bring them up in?"

If I had a fourth-grade daughter, and the choice was rearing her in (a) a stress-free environment, or (b) an environment with some stress in which she learned how to use a ruler, I know which one I'd pick.

Meanwhile, over in Boston, the educators insist that the "failing" labels are wrong:

...For the first time, a district could land on the federal watch list if just a single category of students fell below federal standards. Educators attributed the ballooning list to that new provision, which isolated the performance of groups including Hispanics, blacks, special education students, and low-income students...

"To me, it's sort of like the terrorist alerts that the federal government puts out and says, 'Everybody watch out, there's terrorists out there," he said. "No Child Left Behind is trying to motivate people from fear rather than well-thought-out initiatives."

So is it better for schools not to know if one group is doing more poorly than another? Is it better for parents not to know this? Mayer is apparently of the belief that any shortcoming within a school shouldn't be made public. But others don't agree:

Tom Loveless, a senior fellow at the Brookings Institution in Washington who has studied how states comply with the federal law, said bringing shortcomings to light is likely to spur progress.

"The real question is, is it better to know or not to know" how groups of students are doing, he said. "I think there is a growing public awareness it's better to know."

Mind the Gap! (in Minnesota)

There's much test-score fretting in the North Star state:

Minnesota students are traditionally among the nation's top performers on key standardized tests. Unfortunately, the statewide averages mask an embarrassing reality. Students of color consistently score far below their white classmates.

A recent report from the Education Trust, Inc., highlighted the issue. Minnesota eighth graders ranked first in the nation in math on the 2003 National Assessment for Educational Progress. The average score among the state's white students (291) topped the list. The average score for African American students in Minnesota (251) ranked 22nd among the 50 states. Only Wisconsin had a wider gap between white and black scores.

I can't find the report on the Ed Trust site, although there's a lot of other good information on there. I particularly liked this article entitled "Good Teaching Matters," although that's another "duh" statement as far as I'm concerned.

But I digress:

The low test scores are a point of frustration to some; a source of anger for others. The Rev. Randolph Staten of the Minnesota Coalition of Black Churches says state officials have failed to adequately address the educational disparities.

Achievement gaps are often attributed to income level and home environment. Low-income families often have few educational resources at home. Recent immigrants don't always have the English language skills needed to keep pace in school. Some experts also point to low classroom expectations, peer pressure and teacher quality as key factors.

Nice to see that the tests aren't vilified here. And few reporters will touch upon the hot button of peer pressure and testing, even though at least one study suggests that peer pressure is more highly related to test score performance than is family income. It's more PC to blame the tests than to blame the negative peer pressure and low expectations that abound in poor schools.

Anyway, I tried to find out more about what's being discussed, and done, in Minnesota. (Note to self: Avoid future Google searches using "Minnesota score gap" as keywords, since this produces an avalanche of Packers articles.)

I found some 2003 NAEP data which suggests that the gap between fourth-grade boys and girls is increasing in reading; on the other hand, the black-white gap decreased slightly in fourth-grade math. Eighth-grade gaps between black and white students did not appear significantly changed from the previous year - which is good, because they're wide in both math and reading. If anyone knows of other articles that examine the Minnesota gap, let me know.

Worries about the new SAT and "that essay" unfounded

There's a new "new SAT" article making the rounds. Let's examine it, shall we?

The SAT is undergoing significant changes in 2005, including the elimination of those dreaded analogies...

Hey, I liked those!

...and the addition of a Writing section that includes an equally dreaded 25-minute essay...

Students who plan on attending college should NOT be afraid of having to write a short essay in 25 minutes. This is hardly setting the standard too high.

The changes are:

_A Writing section will be added, with questions on error identification, sentence improvement and paragraph improvement, plus a 25-minute essay. The writing test replaces the SAT II writing test previously taken by students applying to selective schools.

The revisions to the exam, the first in 10 years, make the test "better reflect what students are actually doing in classrooms," says Kristin Carnahan, associate director of public affairs for the College Board, the organization that designs and administers both the SAT and PSAT.

At the same time, some observers began saying cultural bias in questions' wording hurt the scores of minorities.

Not a bad timeline. Actually, the article in general is very good, and even-handed. The only "critics say" line is above, where it's actually appropriate. The reporter spends a lot of time on the topic that is most nerve-wracking - "that essay":

As for that essay - it will be read by two graders in a process that has been followed for years by the College Board in grading its SAT II writing test. Furthermore, the essay counts for only one-third of the Writing grade.

So students should relax.

An interesting testing dilemma

Jason Karlawish, an assistant professor of medicine at the University of Pennsylvania, wonders if a standardized test would be useful in judging who is competent to vote in elections. Needless to say, this has some folks in a tizzy:

When should people with Alzheimer's or other cognitive impairments lose the right to vote? A new report suggests it's when they can't pass a standardized competency test.

The recommendations appear in a commentary in the current issue of the Journal of the American Medical Association. Jason Karlawish, an assistant professor of medicine at the University of Pennsylvania, said the panel decided to tackle the issue of voting rights for the mentally ill -- especially those suffering from senility -- after reading online posts from caregivers of Alzheimer's patients. The caregivers were discussing the 2000 presidential election and the voting habits -- if any -- of the people they cared for.

A test to answer the questions would make sense in situations when someone is being put under guardianship, Karlawish said. Tests could also be appropriate in places such as nursing homes, he added.

Dr. Karlawish is active in research in geriatrics, aging, bioethics and Alzheimer's, so I'm betting his heart is in the right place. He - rightfully, I think - sees testing as a way to avoid wholesale bans, such as the one in Maine which grouped all "mentally ill" voters into one category (one wonders how they defined that group). I wasn't able to find the test in question here, although on other websites related to his research, I found information on other cognitive exams that he's recommended. He's a pro-psychometrics guy, from everything I can tell.

And what's the opposition's argument? The one lawyer quoted above resorts to hyperbole instead of reason - but then presents some sounder ideas:

Mathis, the legal advocate for the mentally ill, doesn't like the idea of a test, especially if it's administered by nursing home staff members. (The report suggests that election officials could take on the role.) "This type of test is extremely subjective, and it's fraught with the potential for abuse," she said.

A better approach, she said, would be to bring up the issue of voting competence during hearings to determine whether someone should be placed guardianship because of mental illness.

Agreed. No one other than disinterested and well-trained parties, such as psychometricians and clinical psychologists, should administer and score the exam. And certainly hearings would be a suitable place to discuss voting competence - although there's no reason that objective measures couldn't be introduced, and weighed, at such proceedings.

Parents, students, and the TAKS

In the Lone Star State, parents are getting their hands dirty by taking on TAKS tutoring duty:

Learning the skills to master the Texas Assessment of Knowledge and Skills test does not have to begin and end with the classroom. Parents can play a major role in how successful their children are with the test by practicing at home, and the Ector County Independent School District Family Education Center is trying to help.

Barbara Villaloboz, ECISD parent involvement specialist, said the education center wants parents to encourage their children to ask questions and to reason...Villaloboz said parents can help their children to do well on the test by teaching them how to read labels on products and asking questions about the weather and grass.

A useful message to send to parents. However, others are ready to give up, claiming the test is too difficult - at least when used as an exit exam:

A revised scoring plan designed to boost passing rates on the new high school graduation test was pitched Thursday by State Board of Education members worried about the possibility of massive failures on the exam.

That would allow students to offset a failing score in one area with high scores in one or more of the other subjects so they could pass the exam.
Not all the board members liked this proposal, though. The Class of 2005 will be the first to be required to pass the new exit exam for graduation. They have five chances to pass, and if this composite plan is approved, Students take the test in spring of their junior year and have five chances to pass before the end of their senior year. Yet some are concerned about passing rates - and increased standards:

Results published this week by the Texas Education Agency indicate that 78 percent of high school seniors have passed after two rounds of the exam. That leaves almost 50,000 students who haven't passed. They have three more opportunities in their senior year. Students have done best on the social studies section, with 98 percent passing.

Juniors this school year will have to get more answers correct to pass, and the cutoff score will increase again for juniors in the 2005-06 school year. The test replaced by the TAKS was much easier, measuring only eighth-grade skills.

So people are concerned that 12th-graders now need to display above-8th-grade skills. Hmm. Take a look at the new exam for yourself - here are the exit exams for the English, math, science, and social studies.

The 98% passing rate on social studies intrigues me. The Spring 2004 results are decent, with 85-87% passing overall on the other three components. and the Social Studies rate is high acros the board. Is the passing standard set lower for the social studies segment? Is the passing rate on English lower than social studies because the English portion has a writing exam? Interesting.

BTW, kudos to Texas for the testing site, which is easy to navigate, easy to read, and stuffed with information about all of the state exams.

He's not one to talk about low verbal SAT scores

I can only hope that the principal of Monroe Area Comprehensive High School is misquoted in this article about using the SAT to assess educational progress in Georgia:

While Walton’s system-wide Scholastic Aptitude Test (SAT) scores showed a drop-off from the previous year in most instances, local officials are questioning the state’s use of the SAT as a barometer for education.

“Statistics dictate that the more folks that take the test, the more you’re closer you’re going to get your mean score,” said Dr. Jimmy Stokes, principal of Monroe Area Comprehensive High School.

Whaa? What? Even if you try to figure out where some words must have been accidentally inserted, that doesn't make sense. Yes, if more lower-ability students took the test, the mean might drop. If a larger number of students who might not otherwise be considering college took the test, that might do it. But it could happen that more students whose abilities range from high to low were added this year, and they aren't going to shift the mean that much. "Statistics dictate," my ass.

Stokes follows this mush-mouth comment with one that is crystal clear, and wrong.

The mean score for the SAT is 800, Stokes said.

The top possible score per section is 800. The national mean for 2004 on the two sections combined is 1049. Reporter Joe Dennis should learn to use Google.

And the Monroe Area Comprehensive High School should find a better spokesperson. Especially when the article immediately follows his claim with an example of a system which has an increase in SAT-takers and an increase in mean score:

Monroe Area saw a 19 percent jump in the number of students who took the SAT. Despite the jump, the school actually saw its math test scores improve, from 447 in 2003 to 452 in 2004. It’s verbal score dropped from 466 in 2003 to 445 in 2004 for a total score of 897.

That would be "its," not "it's." Grrr.

The article also includes this interesting, and unchallenged, statement:

While the state places a heavy emphasis on students taking the SAT for college admission, the Walton system is starting to promote the ACT to its juniors and seniors.

“The SAT is not for everyone,” said Dr. Karen Rutter, technology and career coordinator for Walton County Schools. “The ACT is geared more towards what you know, but unfortunately Georgia has pigeon-holed everyone into taking the SAT.”

And, the SAT is about....who you know? Yes, it's true that the ACT differs from the SAT, and the ACT is most definitely based on what kids know:

The ACT is curriculum-based. The ACT is not an aptitude or an IQ test. Instead, the questions on the ACT are directly related to what students have learned in high school courses in English, mathematics, and science. Because the ACT tests are based on what is taught in the high school curriculum, students are generally more comfortable with the ACT than they are with traditional aptitude tests or tests with narrower content.

But the correlation between ACT composite and combined SAT scores has been found to be a cool .92, which is very high for the social sciences. This suggests that (a) the SAT and ACT are tapping into the same skills, and (b) it is unlikely that there are vast numbers of kids who bomb the SAT but would do just fine on the ACT.

Luckily for this school district, someone with sense is employed there :

Superintendent Dr. Tim Lull said the way the educational system is set up in the state doesn’t adequately prepare students for the SAT. “If the push in Georgia is going to be the SAT, then the state has to modify its testing and curriculum,” Lull said.

Lull noted that the Georgia Department of Education uses state-produced tests throughout the year — such as the Criterion Referenced Competency Test and the Georgia High School Graduation Test — to track student progress, but then emphasizes the results of the SAT, a nationally standardized test. “You can’t hold kids to the state standard and then expect them to hold to a national standard,” he said.

He has a point. If the state cares about SAT scores, then it should focus on those skills. It's not a given that the CRCT and GHSGT measure the same skills as the SAT/ACT. It would certainly provide some criterion validity evidence for those tests if they did, though; if in fact large numbers of students are passing the GHSGT but bombing the SAT, that's something for the admins to think about.

When the SAT is just perfect

There's nothing like a group of students doing well to change testing criticism to testing praise, as evidenced by this article that's just chock-full of educators singing the praises of the SAT:

Ohio's class of 2004 achieved scores 30 and 24 points above the national average on the verbal and math Scholastic Aptitude Tests (SATs), according to results released today by the College Board's Advanced Placement Program. Ohio students had an average score of 538 on the verbal test, compared to the national average of 508. They also averaged 542 on the math test, compared to 518 nationally...

"Today's students need higher order thinking skills to compete in a 21st century global economy," Zelman said. "Challenging academic standards and coursework will prepare all of our students for success in college and careers."

Why, there's not a "Critics say" line in sight. Imagine that. From now on, if I see anyone bashing the SAT as measuring nothing but "test-taking skills," I'll just send them along to Zelman, and she'll take care of 'em.

And do a Google search for "national SAT scores" to read all the latest articles about schools that are either celebrating or fretting over their comparisons to the SAT and ACT averages. However, none of the articles link to the College Board site that has a plethora of SAT information - for shame, mainstream reporters.

The gifted-vs-remediation tradeoff

From Pennsylvania comes this tale of students who have learned facts but don't really understand the material - or so the teachers claim:

Wolfe, an eighth grade reading teacher at Big Spring Middle School, helped design the Reading Increases Students' Excellence (RISE) class to prepare students for the ninth-grade standardized writing test...

They had memorized the definitions for a test in other classes, but "they didn't actually know that they had to know it for knowledge, for life."

This, of course, is the lead-in for a crop of test criticism. But I see this as a criticism of their former teachers. What did they do in their language arts classes? Say to the kids, 'Okay, just memorize this list, but you'll never use these words again - they aren't important"? How bad do teachers have to be for kids to get the idea that the English language is not knowledge they'll need for their lives? Don't go blaming tests for this, nuh-uh.

Getting the right answers on a state standardized test is a "game," says Donna Benson. It's a game many of her students refuse to play.

Benson teaches gifted students at Cumberland Valley High School, and says highly intelligent students tend to ignore test-taking conventions, especially when writing essays. Instead, many write creatively, and as a result score low on the Pennsylvania System of School Assessment (PSSA).

I hear this a lot, but I've never seen any evidence to back it up. Anecdotal evidence, sure. But there isn't a scoring rubric out there that doesn't give examinees a boost for using correct spelling, grammar, and vocabulary. Is it really true that there are vast numbers of very smart kids whose writing is so creative that, despite proper vocabulary and spelling, their essays recieve very low, or flunking, grades?

"It's done kind of as formula writing," Benson says of how the writing portion of the PSSA is scored. "I want the kids to know what the formula is, but ... I want the kids to go beyond that," she says.

And they can. But they should understand why the formula is there, and why it would be silly to protest the formula on the grounds that it's too "dumb" for them. For a lot of kids, it would be a big step forward just to be able to write well, period, even if the writing came out formulaic.

Highly intelligent students often have trouble with multiple choice and true-or-false questions because they "over-analyze" the question, Benson says. She worries gifted and bright children get left behind when schools emphasize remediation...

"I'm glad my children aren't in school anymore," says Candy Shively, who works for Cumberland Valley School District and used to teach special interest classes. "I think we're really skimping on the enrichment things and the higher level of thinking."

And I can understand her way of thinking. Unfortunately for the bright kids, the act is called Leave No Child Behind, not Push The Smart Ones Ahead. Proficiency isn't enough for a subset of kids, true. But when so many students are failing miserably, schools are often forced to focus on them.

Harnessing the power of the web

From Carnegie Mellon comes this advancement in education:

Carnegie Mellon has developed a Web-based computer tutoring system to help middle-school students prepare for standardized mathematics tests like those required under the federal No Child Left Behind Act. The "Assistment" system aims to solve a dilemma for teachers: how to prepare students for tests without sacrificing quality instruction time. The system is designed to quickly predict a student's score on a standardized test, provide feedback to teachers about how they can adapt their lessons to address students' problems and provide individualized tutoring to suit each student's needs. The system is being tested in Massachusetts with a grant from the U.S. Department of Education, but it can be easily adapted for use in other states. In developing Assistment, researchers have drawn upon the proven success of Carnegie Mellon's popular Cognitive Tutor®, a comprehensive secondary mathematics curricula and computer-based tutoring program that is in use in 1,500 schools nationwide. Contact: Jonathan Potts at 412-268-6094 or Anne Watzman at 412-268-3830.

I found more info at the Pittsburgh Advanced Cognitive Tutor site. Sounds interesting.

Useful info for CA parents

The Contra Costa Times has an amusing round-up of info for parents whose eyes glaze over when confronted with stacks of percentile-ranking charts:

There's no nip in the air and the leaves are a steadfast green, but the shift of seasons is palpable.

Frankly, there's enough school assessment data flowing from Sacramento this month to make any parent's eyes glaze over. Cucamonga kids performed how well? Inyo County fifth-graders posted what percentages? School assessment data has been available online for some time, but in a format only an education policy wonk or closet statistician could love.

I never knew there were statisticians in the closet. Maybe that's because I'm surrounded by people like myself who are the statistics-geek equivalents of the drag queens parading down 5th Avenue during the Gay Pride parades. We love numbers, and aren't ashamed of it.

Telling it like it is

Miriam K. Freedman, whose work I've covered in N2P before, has a great website and a new article at the Hoover Digest about the fight for high standards. It's phenomenal, and you should all read the whole thing, but if you don't have time, here are a few of my favorite spots:

High-stakes tests, which affect a student’s ability to earn a high school diploma, are now in place or on the drawing boards in about half the states. Often controversial, they have spawned “test boycotts” and lawsuits...But a strange thing is happening: As we get closer to having the graduation tests “count,” many leaders have blinked, with the result that standards are compromised and test results invalidated...

Blinking at standards fails the public when, as the going gets rough, we quietly alter tests and standards. Such compromises have a crippling ripple effect on education reform, leading to cynicism and loss of faith in the entire venture. Such compromises are not legally warranted. For the sake of our children and for America, we must not blink.

Laptops are not the answer

For those who may have been hoping that laptops were magic wands when it came to test scores, these results are surely a disappointment:

Middle school students who used laptop computers for two years performed about the same on a standardized test as students in the past who did not have access to computers. Critics of laptops say the scores are the first real evidence the program, which has cost the state more than $15 million, is an expensive fad. Proponents say it is too early to expect dramatic changes in test scores...

Scores for reading, writing, math and science in the Maine Educational Assessment were essentially unchanged in the past school year among eighth grade students compared with the previous two years. But there was a measurable improvement in writing scores among students who took the online version of the test at 60 schools...

Do know (a little) somethin' bout history

Florida's students are improving in history, as measured by standardized test scores:

About 86 percent of students taking U.S. history passed a new districtwide standardized test this spring. About 74 percent of world history students passed a similar exam. Two years ago, when the 100-question tests were first implemented, so many students were in danger of failing that passing grades were curved to the extreme — a student needed to answer just 23 percent of the questions correctly.

"It's been met with some resistance," Palm Beach Central High School teacher Don Meyers said. "But you have to adjust. That's the reality of education in the age of accountability."

Yes. Yes, it is. It's a reality in a day where people are starting to acknowledge that it's a travesty when students who have taken US History can't answer 23 questions out of 100 correctly. And the details provided in the rest of the article provides clues as to why things might have gotten so bad.

In American history, teachers have one year to cover from 12,000 years ago, or prehistory, to the current Iraq war. It was easy for teachers to focus on one era they were particularly interested in and gloss over others.

12,000 years? I know they had to cover the Native Americans, but surely no one thought that any substantial percentage (more than 10% or so) of American History class time should be spent on the many, many years before America was America.

School board member Debra Robinson provided the impetus for the standardized history test because of her concerns that African and black history were not being taught. The test allows administrators to see whether teachers are touching on key areas.

For example, in American history, students did the worst on questions on the effect of immigrant groups and questions on political events in contemporary America. They did the best on multicultural questions and black history questions, getting an average of 77 percent and 79 percent, respectively, of those correct.

Well, then now we know what students need to focus on - immigrant groups and contemporary politics.

In world history, students struggled with the Age of Reason and the Industrial Revolution. They did the best on the Age of Discovery and, again, on African and black studies.

I'd say the Age of Reason and the Industrial Revolution are fairly important topics.

Still, Robinson said she continues to hear anecdotally that the black curriculum is not being taught in the detail it deserves.

Yes, but the data suggest that's not true. Wasn't the point of this test to see where students need extra help?

She's also concerned about the average scores on the tests. Although 86 percent of students passed the American history exam, the average score was 75, a solid C. In world history, the average score was 67, a D-plus...

"It's sad when you say 70 is good," Robinson said. "I think that shows what a pitiful state of affairs we're dealing with in general. The definition of pass in my house is a B. I won't kill you if you bring home a C, but I'm not happy either."

I agree. They may be passing now, but just barely.

Spinning the scores in New Mexico

Things aren't looking great in New Mexico:

Barely half of New Mexico’s high school juniors were proficient in reading and less than half were proficient in math, according to results of a standardized test released Friday by the state Public Education Department. The first-time assessment test, taken by more than 19,000 students in November, will be used to rate public high schools. Results show 56 percent of students are proficient in reading and 47 percent are proficient in math...

Clovis schools Superintendent Neil Nuttall praised the test as a fair assessment.
“We feel OK. I don’t think we can say we feel good, but what makes us feel more encouraged more than anything is what we have put into place,” Nuttall said. “We have beefed up our curriculum. We have beefed up courses. We have increased the number of requirements for graduation.”

Is there some course that superintendents take in college that teaches them how to put a positive spin on anything, including the news that half their students are not proficient in reading and math? If so, it looks like a few administrators were absent the day that spin was taught in class:

Portales schools Superintendent Jim Holloway did not return phone calls placed to his home and office. Portales High School Principal Melvin Nusser did not return calls placed to his home Saturday.

But one educator is a master of the "spinning by redefinition" technique:

Fort Sumner High School ranked above state average in both subjects. Last month, Superintendent Lecil Richards apologized to state officials for failing to administer a standardized test to fourth and eighth graders in March. He said he will leave it up to the state to decide if Friday’s results vindicate him and prove his schools are excelling.

“Test scores are just one indication and not the only indication of how our children are doing,” he said. “I really look at how they do in life, how they handle themselves and how they do after high school.”

Because, as we all know, students who can't read and don't have mathematical skills are likely to succeed after high school, thanks to how well they "handle themselves."

Fearing the worst

An article about political response to Illinois' recent cutback of standardized tests has some telling comments. Some legislators are horrified about the cuts, and it's worth noting why they fear this will have a negative impact on education:

A decision by state legislators to scrap social science and writing exams in Illinois grammar schools is drawing mixed reactions from local educators.

Norridge District 80 Superintendent Sue Knight is dismayed. "I was absolutely speechless," she said. "I couldn't believe this was happening. It's a blow to all of us who are dedicated to doing the very best for our children."

"Now, our kindergarten students know what a complete sentence is, and our students know how good writing is done."

The teachers were aghast that third-graders would have learn to write paragraphs? How did things get that bad in Illinois? If the teachers weren't expecting third-graders to write in complete paragraphs, I bet you dollars to doughnuts that the teachers didn't expect these kids to read complete paragraphs, either. How on earth did they manage to take three full years of school just to get a kid up to the point of being able to write a complete sentence? No wonder Ms. Knight is afraid that removing the test will have a negative impact.

The standardized supernatural

The fear of standardized tests is apparently so strong that references to testing bugaboos have crept out of news reports - and into the movies:

When Winter Garden novelist Edward Bloor set out to write a chiller for young adults, he decided to include a murderous poltergeist. Then he added a really scary touch. The poltergeist lives in a magnet school where the only thing that the students do, day after day, class after class, is take standardized tests.

It's the spookiest fictional school setting since Carrie made a mess out of the prom.

Hee hee hee. I think it makes for a nice surreal (and frightening) touch.

"I just thought it would be humorous to take things to the nth degree -- but then, I have an extreme sense of humor," [Bloor] says. Perhaps. But he's not alone. The hopes and fears of many students, from grade-schoolers on up, are so intertwined with standardized testing that references to it have crept out of headlines and PTA meetings and into the province of popular culture. Novels, movies and television shows are full of young characters grappling with issues related to test scores...

Not even the reigning prince of young-adult literature is exempt from testing-related tremors: The title character of Harry Potter and the Order of the Phoenix gets the shivers over his upcoming exam, the O.W.L. -- Ordinary Wizarding Level.

Oh, rock on. That's too awesome. Not even the soon-to-be most powerful wizard in the world will be able to escape the horror of --- STANDARDIZED TESTS. Tee hee hee.

I have no problems with testing critics who have a sense of humor, and are willing to show us how absurd testing (like anything) can be when taken too far. Kudos to Bloor. But I bet you some humorless educrat will watch this movie and try to convince the public that some schools, and tests, really ARE this bad.

The writing test gets written off

Illinois' new budget cuts mean the state standardized exams in writing and social studies are history. Some educators have no problem with this, claiming the tests were "inessential":

"Standardized tests are important, but they're nonessential in 85 percent of the school district. In the state of Illinois we are in deficit spending right now. We're in a nationwide crisis and non-essential programs are going to have to go," says Guilford High School history teacher Walter Hoshaw...

Hoshaw says there are plenty of other ways to test a student's ability. "We have pretty good measures through the SAT and ACT tests that we do (with) our high school kids. That gives us a much better measure of how our kids are doing," says Hoshaw.

Others, though, aren't so sure:

The decision was as much about the state's testing philosophy as saving money. It is not expected to change even if the money were restored.

"I have heard the philosophical argument that if we stop testing it, schools will stop teaching it. But I believe in professional educators more than that. We have to remember that, long before this standardized student movement, teachers were teaching their students how to write."

Yes, they were. But are they still?

The announcement already has educators rethinking their priorities for next year. Becky McCabe, a principal in Urbana, said she originally planned to make writing part of her school's improvement plan. But the testing change means her staff likely will refocus on improving reading skills.

"I hate to say this, but you treasure what you measure," McCabe said. "When it comes down to money, you're going to focus on things that are on the bubble. Writing is just not going to be the same, and that's a shame."

Have we really reached a point where teachers - teachers, for heaven's sakes - won't teach kids how to write in any non-tested fashion? Where do the teachers think kids will pick this up, if not in school? Comments like this make me wonder if Representative Eddy really understands the lack of focus on writing skills these days.

Update: Devoted Reader JW has this to add:

A liitle more background for your post about dropping the Social Studies test here. The most important thing to understand is that Illinois is undergoing a profound, severe, and probably long lasting budget crisis. There are a lot of reasons for this that don't matter much for this discussion. Every state program is getting chopped to some degree. The Social Studies test doesn't seem to measure very much, or to provide useful info for parents. Districts that do well on the reading and math tests tend to do well on the SS, so it gets the chop. Sure, there are a lot of teachers and administrators who aren't nuts about NCLB, and who don't like testing anyway; it's part of the territory. There are also a lot who just deal with it without being obsessed one way or another-like, I'm happy to say...

Democrats and teachers unions

The dance between soon-to-be Democratic presidential candidate John Kerry and the educational unions ought to be fun to watch over the next 100 days...

...while education experts agree that Kerry has mapped out a more liberal education agenda, he has surprised some educators with more moderate proposals.

"I believe we need to offer teachers more pay. More training, more career choices, and more options for education. And we must ask for more in return, that's the bargain," Kerry said.

Emphasis mine. Think this was because Kerry sensed the parental approval of get-tough acts like NCLB? Whatever the case, he now appears to be backpedaling. So he's for "pay for performance," until he's against it. And even though Kerry's not yet at the convention, the teachers are, in full force:

Milton Bond Jr., a high school teacher from Milwaukee, Wis., and a first-time delegate from that state, said he wanted to come to the Boston convention because of his concern about the direction the country is heading under President Bush...

“I feel like you’re punishing schools, and you’re punishing students. It’s an empty policy,” he said.

But we are punishing schools. That's the point. We're trying to punish the schools that are punishing students by depriving them of a quality education. It's pretty hard to hold schools accountable for anything if there's never any punitive action for the schools who fail their students. Perhaps Kerry, with his talent for "nuance," can think of some way to get accountability with only praise, never punishment.

And then there's this:

Convention delegates flipping through copies of the Boston Globe on Monday might have stumbled upon a provocative quarter-page advertisement with the headline: "No Child Left Behind?" The ad, signed by more than 100 classroom teachers, parents, noted education advocates, and others, suggests the federal law is part of a plan by President Bush "to privatize America's public schools," and that it threatens thousands of schools with closure. The law, the ad argues, encourages "lying about the facts" and "uses blacklists to banish professionals, institutions, methods, and books."

"It's outrageous," said Andrew J. Rotherham, the director of education policy at the Progressive Policy Institute, a Washington-based think tank aligned with the centrist Democratic Leadership Council. "There are legitimate criticisms of No Child Left Behind, but that ad seems to go out of its way to avoid them."

Yes, it does, and in doing so overplays its hand. I can't improve upon Mr. Rotherham's summary of the effect of the ad:

...Mr. Rotherham suggests that the ad's rhetoric may well undermine its mission.

"Ads like this hurt the cause of people seeking changes in No Child Left Behind, rather than help it," he added. "Your average person sees an ad like that and is going to smell weirdness, not reasoned debate."

But Mom, I have to have this new dress from the Gap! It'll help me on my SATs!

Whee! Let's have fun! Let's get kids wondering whether the outfit they wear on the day of the SAT will effect their score! That's so much more fun than drilling them on vocabulary!

Test-prep company Peterson's says it plans go beyond drilling students in the three Rs, starting what it's calling a testing laboratory to see whether students gain any edge on the SAT from the little things - the choice of pre-exam meal, the hue of their clothes, the music they hear on the drive to the test...

"We don't want people to think they really will raise their scores 100 points if they wear the right color," said Jessica Rohm, vice president of communications for Thomson Learning, Peterson's parent company. But "just taking the edge off by bringing in some fun things associated with testing I think will raise their scores a little."

"Fun things associated with testing"? C'mon. I'm in testing, and "fun" is not one of the first five words I'd associate with the field, or with the image the public has of it. I'm all for taking the edge off for examinees - test anxiety is through the roof these days, in no small part because of inaccurate press which claims that tests are biased, unfair, useless, or all three. But I don't think everything should have to be "fun" to get students involved, and that includes a demanding admissions tests.

I'll eagerly await the outcome of Peterson's experiments, but I'm betting the top scores will continue to be delivered by students who see the SAT as a challenge for which one has to methodically and intensely prepare, and are able to motivate themselves for it without "fun" - or wardrobe considerations (how classist! What about those kids who don't have that many clothes to wear? And doesn't that just add more stress for obsessive clotheshorses?) - being in the picture.

Testing brouhaha in New Hampshire

Well, well. There appears to be a clash between teachers and SAU administrators in New Hampshire's North Hampton school district:

The School Board voted not to test North Hampton School students next year using a standardized test recommended by SAU 21 administrators. The NorthWest Educational Association tests were used this year, but teachers at NHS unanimously said they believe the tests are not useful and should not be continued. School Administrative Unit administrators suggested that teachers at NHS were not trained to properly use the tests and so cannot judge their effectiveness.

Ouch! But it's worth asking - who was in charge of making sure the teachers were properly trained? The NWEA is a computer-adaptive exam, so the students would have needed a bit of training as well.

NHS Principal Peter Sweet said teachers "tried to make (the test) meaningful," adding, "They used and shared the data, but they don’t want to do it again." Sweet said he would prefer that the school focus on grade-level assessments developed by the teachers to monitor students’ learning.

Teachers have complained that because of the structure of the test, students might end up being asked questions usually meant for much-older kids, and this kind of data is not helpful for how they want to teach.

SAU administrators pointed out that other districts in which more teacher training took place seem happier with the test, and more able to use the scores.

Too much work for too little "growth"?

So far, we're not seeing a lot of good results from Florida's summer camps intended to boost FCAT passing rates:

Low grades and failing FCAT scores sent thousands of students back to classrooms throughout Southwest Florida this summer. For third- and 10th-graders who had to pass the FCAT or the alternative SAT-9 to advance, few were successful.

"There's not one person that we didn't see growth in their skills and that's what's important," said Michele Markstahler, lead teacher for the summer program at Sallie Jones.

FCAT items and test prep

In all the hubbub of commenting and contradictions on last week's post on the infamous geometry question posed to Governor Bush, I completely forgot to go find what Charles of Reform K-12 had to say on the topic. An unforgiveable mistake, on my part.

Charles is not only a superb geometry teacher, but is more willing to go out on a limb than I. While I tactfully said that I didn't think the item mentioned by Ms. Marques was on the FCAT, Charles said, "We have no doubt that there was some question involving either a 3-4-5 right triangle or a 30-60-90 right triangle, but not both. We will publicly eat a copy of the FCAT if anyone can prove us wrong."

Like me, he also found it appalling that it was "news" that the Governor got the item wrong, but not that the girl herself got it wrong:

No one expects grownups in non-scientific fields to remember high-school geometry, so the Governor's ignorance is understandable. But here is not only one student, but her friends as well, who are confoundingly ignorant on basic facts from 10th grade geometry, while being convinced they know their stuff.

In other FCAT-related news, I received an email from Devoted Reader, and concerned parent, V.G., who is an FCAT supporter but has some questions about a "research" FCAT remediation class. It seems that schools receive funding to place students in this remediation class and may be, um, overeager to place students (even those who are performing at grade level) in the class. Has anyone else had an experience like this?

Why is there a flashing red light on my kid's test score report?

California has decided upon a traffic-light theme to make test scores more interpretable to parents - probably not a bad choice in a state that's as car-crazy as California:

Parents of California's public school students can expect easier explanations this summer of how their children are performing on state standardized tests. Education officials have scrapped complicated old report cards that they said confused parents and replaced them with easier charts and color-coded guides.

Students will also gets lists of their strengths and areas that need more focus. For the first time, the reports also will include translation guides in Spanish, Chinese, Hmong, Korean, Tagalog and Vietnamese.

You can't handle the truth!

First it was a secret. Then it wasn't. Now it is, again. Georgia's DOE needs to make up their minds:

The state will delay going public with any more "cut" scores — the number of correct answers needed to pass a standardized test — until the Board of Education adopts a formal process for releasing the figures.

Last month, the state released cut scores for the Criterion-Referenced Competency Test following an open records request from The Atlanta Journal-Constitution and a unanimous board vote that the cut scores should be released. The newspaper also requested cut scores for two other standardized tests: End of Course Tests, taken by high school and some middle school students, and the Georgia High School Graduation Test.

Basically, they're not revealing any more cutscores until the public has the information they need to interpret those scores correctly (and not make disparaging comments about how low the cutscore is, as some other journalists - and I - have done). It's certainly not a bad idea to make sure a definition of cutscore, along with other test information, gets released when the cutscores do. So let's see what Georgia ultimately releases, if anything.

Even a test score will bounce, if you drop it from a high enough platform

What's going on with test scores in Delaware? And where's Dave Huber on this?

Buoyed last month by an apparent bounce in the state's standardized test scores, elected officials rushed to put themselves in the way of reflected glory...

Nonetheless, both Minner and Education Secretary Valerie Woodruff said the improvement in test results is genuine.

The rest of the article, which is long but worth reading, goes on to explain that when the repeaters eventually reach the 10th-grade exam (next year), the scores will most likely drop down a bit. It's true that it's hard to disentangle the real progress from the examinee pool anomalies in Delaware's startling test score improvements (53 percent met the math standard, compared with last year's 45 percent; 71 percent met the reading standard, compared with 66 percent last year). The retention rate for eighth-graders in 2002 was 11.5 percent (as compared to 3.8 percent in 200), so that's a sizeable little chunk that didn't take the 10th-grade exams this year.

Of course, if retaining those kids did in fact help them to be better prepared for when they do take the test (results so far suggest otherwise), the test scores won't drop as much. So I think next year's scores are going to be very informative.

Me, I'm just savoring the opportunity to read a news article in which it is assumed that a state's standardized exam scores are accurate indicators of true student ability (we're seeing high scores because the examinees are smarter, for whatever reason). Funny, but don't we almost always read the opposite - tests are biased, educators shouldn't assume that student ability is being accurately portrayed - when low test scores are observed? Why isn't this reporter rushing to remind us that "critics say" such tests are biased? Most times, we see quote after quote in articles trying to convince us that the tests are meaningless and that students are really doing much better than scores would suggest; here, we're being cautioned to think that students are actually doing worse than scores would suggest.

This reminds me of a time-honored technique of testing critics, in which they damn all tests as biased and unfair - unless those tests support a political point they want to make. Witness the groups who claim SAT scores are biased, but then use SAT scores to make a point about a different test.

Gives new meaning to the phrase, "high stakes"

You know, Governor Bush should have expected something like this:

During a speech to high school students who mentor younger children in reading, a teenager asked the governor a basic geometry question taken from the Florida Comprehensive Assessment Test, which Bush has championed.

"If the point is, I haven't been in school for the last 30 years, that's true. But if I'm going to be graduating from high school and I can't pass a 10th-grade aptitude test, then I'm fooling myself," Bush said. "The fact that a 51-year-old man can't answer a question, is really not relevant. You're still going to have to take the FCAT and you're still going to have to pass it in order to get a high school degree."

Marques thinks Governor Bush didn't have much of a sense of humor about the whole thing, and he should have, but to be fair, he could have pointed out the grammatical atrocity in her opening statement, "Me and a couple of my friends...were wondering..." And what if he had known her answer was incorrect and pointed that out to her as well? Why did she ask a question to which she mistakenly thought she knew the answer?

BestOftheWeb points out:

Now first of all, does anyone who isn't a graduate student in mathematics know the answer to this? We certainly didn't. Besides, geometry is the most useless branch of mathematics, at least in our experience. We occasionally make use of algebra, trigonometry and calculus, but we dropped our high school geometry class after a couple of months, and we've never missed it.

I certainly wouldn't argue that geometry was useless, but I wouldn't have known the answer to that item off the top of my head, either, and I aced AP Geometry in high school, and I'm younger than Governor Bush. While I may not remember the proofs, the mental discipline that came from doing them has helped me in many ways unrelated to math, and to suggest that students shouldn't learn that material just because Bush doesn't know the answer to one item off the top of his head is just plain silly.

Update: Given the discussion in my comments section, I think the theory floated by commenter Josh - that the young lady misremembered a 1-2-sqrt(3) item as a 3-4-5 item - is quite possible. According to the FCAT math item specifications, the only trigonometry tested on the 10th-grade FCAT is right-angle trigonometry (the famous SOH-CAH-TOA). It's hard to believe that the uses of inverse trig functions (necessary to solve a 3-4-5) were tested on this exam. Perhaps she misremembered a Pythagorean theorem item that used a 3-4-5 triangle.

Given that WFTV went to the trouble to contact a mathematics graduate student for the correct answer (and note that they didn't ask a Florida high school math teacher), why didn't they ask her what skills were required to solve the item, and compare those skills to the FCAT item specs listed online? Why did WFTV (and I, for that matter) miss the possibility that the student misstated the item, especially given that she gave the wrong answer for the item she posed?

Or was the point here just to have a hook on which to hang two "critics say" claims and convince the audience that the FCAT is bad because Governor Bush was stumped by one item?

The SAT vs. the ruling classes

From Business Week Online, a nice profile of ETS founder Henry Chauncey:

In the early 1930s, Harvard University's graduating classes were made up of young men and (a few) women who had spent their teen years reveling in the heady 1920s. It seemed to one young Harvard assistant dean, Henry Chauncey, that these sons and daughters of the elite were simply expecting that they would rightfully inherit top positions in business and society, as if the Great Depression couldn't touch the ruling class.

Chauncey, a Harvard grad himself, was distressed to watch as class after left the august institution, and failed, in his view, to meet the titanic civic challenges of the times. Together with then-Harvard president James Bryant Conant, Chauncey initiated an experiment to bring Harvard a new type of student, based not on the connections they or their parents had, but solely on what the students knew and their potential for further learning.

Many people would like to forget (if indeed they ever knew) the proud principles upon which ETS was based. Yes, there are still inequalities in our educational system, and students with more wealth and health and good fortune still tend to do better. But it's a far cry from the days where, if you were very smart but from a poor family in Nowhere, Kansas, you were completely out of luck, and tests like these have a lot to do with that.

So, how did you feel about your elementary school experience?

One elementary school in Vermont isn't interested in the state standardized exams, and believes that surveys - yes, surveys - are the way to see if they're really being effective:

In an attempt to measure Marlboro Elementary School's quality without federally mandated exams, surveys will be sent out to graduates from the past eight years. A draft of the survey has been written by Marlboro Elementary alumnus Ariel Poster, who is a sophomore at Barnard College in New York City, a women's college affiliated with Columbia University. She was hired by the Marlboro Elementary School Action Plan Committee to evaluate the school's ability to educate.

The tricky part will be finding all the graduates, said board member Lauren Poster, Ariel's mother. The survey will go to high school students and graduates old enough to be in college.

Oh, this I gotta see. A survey. Are these folks aware that a response rate of 25% is considered normal for surveys? Are they aware that respondents self-select, and that only those who were extremely satisifed/dissatisfied may bother with responding? Are they going to account for the fact that a respondent's recollections and impressions of their time in elementary school, so far after the fact, may not only be incorrect, but also have nothing to do with how well the school did in educating students as a whole? Why is it more important, or more meaningful, to ask former students what they remember of the experience, than to see how current students are doing now?

And how 'bout that "you are an incredible person!" opener? This survey is going to high school and college students. You'd think Marlboro wouldn't feel the need to use the grade-school touchy-feely tone with more mature students. And what if one of those alumni is going through a rough time? If I'd just gotten fired, dumped, or rejected by the college of my choice, some goofy letter that told me how proud my elementary school was of my "accomplishments" would go straight into the ol' circular file.

In May, the school board announced that the school would no longer administer any tests which the principal finds void of educational value, participate in Adequate Yearly Progress as determined by the act, or forward any information to the Windham Central Supervisory Union that can be connected to a specific student's name...

A CD-ROM will accompany the survey, with the top 10 reasons the graduates should fill them out.

A CD-ROM? Well, that'll weed out any respondents who don't have easy access to a computer. And, face it, there's no pressure for students to respond, much less respond honestly. I eagerly await any data that might be produced from this little experiment.

Update: This interview with Windham Central Supervisory Union assistant superintendent Jim Peters contains a very odd quote on this survey topic:

The federal government, he said, has a bigger need to quantify everything in order to assess a school's quality. The method of standardized testing is something one school in his district has taken a blunt stand against.

"It's not surprising that Vermont has embraced portfolio assessment, because it speaks to parents' needs to see examples and samples of what their child knows and is able to do, instead of an annual achievement test," he said.

Stop right there. A survey is not the same thing as a portfolio. The article from Wednesday did not indicate that anything other than surveys were replacing the state standardized exams. A survey is most definitely not an assessment and it does not fall under a collection of work produced by a student. I still think the survey idea is ridiculous, but if Vermont is indeed using some type of portfolio assessment in addition to the survey - something that wasn't made clear in either article - that makes the whole thing seem not quite so ridiculous.

FCAT wailing and writing woes

John Jay Ray at Dissecting Leftism discovered two good education-related posts today. The first, from Dangerous Liberty, is from someone who's been writing about education in Florida for 10 years, and isn't happy with the anti-testing movement:

....Apparently, it is impossible to criticize public education without hating it. I don't hate public education. I want it to be better than it is.

And yet, when we get the wonderful news that of the 230 kids who repeated third grade after failing the FCAT once, 147 passed it, the entire focus is on the 83 who failed the test again and hand-wringing abounds. The concern is for these children's sense of self-esteem, but what self-esteem can they have if they get out of school unable to read?

Why, they can have the kind of self-esteem that some educators believe in, that comes not from accomplishments but from constantly being told you are fine just as you are, and that you always know best, even if you don't know anything. Other people might term that "narcissism," but hey, why quibble over the fine details?

And then we get the story that the “FCAT is frustrating seniors' plans” (May 10). We learn that a student with a B/C average in high school “has yet” to pass the test.

The observation hits ya right between the eyes, doesn't it? If she has a B/C average and still cannot pass the FCAT after two years and four more opportunities to take the test, she evidently did go to school for 12 years of nothing!

But blaming the test is easier, and it makes for snappier quotes.

I understand that any test is going to be imperfect and that there are bugs to be worked out of the system, but we're seeing some real gains happening that moves me to wonder what the true problem is here. Could it be simply that teachers and schools are not used to being held accountable?

Mmmm, I think this writer is on to something.

The second article is by a guest commentator, Tina Blue, on the Irascible Professor's site. After reading it, boy, do I see why she's Blue:

Those of us who teach college English classes are always overwhelmed by the astonishing deficits in our students' ability to get their facts straight, to think clearly and logically, and to express their ideas in language that actually makes sense and that follows the most basic rules of grammar.

Now, Mrs. Colona did not give us "deep" topics to write about. One I remember was "Write about your favorite holiday memory, and explain what makes that memory so special to you." But you know what? Almost none of the college students I have taught since 1972 could write a 500-word essay on that topic in 45 minutes without that essay's being marred by numerous errors or infelicities in one or more of the following areas: grammar and usage, diction, style, formatting, structure, and logic. I know this for a fact, because we often do give such simple topics as the first essay assignment in English 101.

Why, I see the problem right there. They aren't giving "deep" enough topics these days, for those "deep thinkers" with "deep minds" who can't be expected to do well on standardized tests, much less essays. After all, what is self-discipline, structure, dedication, and instant, objective feedback when compared to the merits of having a "deep" mind?

For more essays by Ms. Blue, visit her Teacher, Teacher website. I've read through some of them, and let me tell you, Ms. Blue has NO problems with expressing exactly what's on her mind in language that is grammatically correct and to the point. And I agree with everything I've read so far.

Oh look, a testing critic!

Devoted Reader Erin T. sent along an email in which she expressed her astonishment about an anti-testing site on the web. Why, I never knew such things existed, did you?The author of this website, Chris Carter, cites Wacky Alfie Kohn's books approvingly, and also posted this "press release" to a teacher's e-board, which is how Erin found it:

I have created a website that summarizes the critical literature regarding standardized admissions tests such as the SAT, GMAT, and so forth.

Because, Lord knows, we need more criticism of testing out there, and it needs to be easier to find. I mean, doesn't it get to you how reporters fall all over themselves to print only the good things about tests?

With few exceptions, students who wish to attend professional schools anywhere in the world, or to attend almost any university in the US, are forced to write these tests.

"Write these tests"? "Forced"? I love it when someone restates a requirement as being "forced" to do something, especially when schools vary widely as to how much emphasis they place on test scores.

I think it is our duty as educators to be familiar with the case against these tests.

I would think it was your duty as educators to be familiar with the research and hard data surrounding these tests, and be familiar with the cases for and against testing. Is there any particular reason that educators should have only this one viewpoint?

If you read my article, you will see that these tests have no validity as predictors of actual accomplishment in any field.

That's because SAT scores predict college grades, which aren't necessarily linked to later performance. Thus, the flurry of low to negative correlations between scores and later accomplishment that the author cites in this article are beside the point. The SAT has never claimed to predict success in life, so criticizing it for failing to do so is incorrect. What's more, for someone who goes to a lot of trouble to explain what a correlation is, Carter leaves out a discussion of restriction of range, possibly because to do so would leave the door open for contradiction of his theories.

An extended discussion of restriction of range wouldn't be appropriate here, but to sum it up quickly, a correlation is a measure of how multiple variables co-relate, or co-vary. If one of those variables has restricted variance, the correlation of that variable with any other variable will be "restricted" or lowered (closer to zero). If a variable does not vary, it cannot co-vary.

SAT scores for college students are restricted, because, for the most part, if you have a low SAT score, you don't get in. So, as a hypothetical example, let's say that most everyone who goes to Harvard has an SAT of higher than 1200. That leaves us with scores of between 1200 to 1600 to correlate with some measure of college success, or later success in life. Given that even smart people will screw up, fail to be "successful," or simply choose to stay out of the rat race, it's very possible for a Harvard grad with an SAT of 1200 will do fine, while one with an SAT of 1500 may drop out, or go bankrupt years later. That results in lowered correlations, but it doesn't necessarily follow from this that the SAT is not useful in college admissions. SAT scores tend to correlate with other measures of intelligence, and as long as we believe intelligence affects college performance, then colleges will have more success with high-SAT admittees than low-SAT scorers.

To continue on with Carter's criticisms:

They have extremely limited validity as predictors of first-year college grades;

For what school? Every schools weights the SAT and ACT differently, because those tests hold different predictive validity for different populations. To average over all schools is to again mislead the reader. And the author reports that, "The SAT has the most predictive validity of the tests1, with correlation coefficients ranging from .2 to .5 at most (R-squared ranging from .04 to .25)," as though this were a bad thing. Obviously, he's unaware of just how rare a correlation of .5 is in the social sciences, especially for one test, taken on one day, with a limited number of items. It's such a tiny snapshot of performance that the correlations of .2 to .5 are just amazing.

Does Carter know of any other snapshot that is this cheap, standardized, and quick for schools to use that will have that high of a correlation with college grades?

...and they are biased against women, minorities, and the poor.

Oh, again with the bias. As OpinionJournal likes to point out, if the world were to come to an end, the NYT would print the headline, "Armageddon arrives; women, minorities hit hardest." I have to give Carter credit for trying to define bias in his article, but then he wedges the idea of bias in where it shouldn't belong, here:

Are the tests biased against the poor? Well, it depends on what you mean by "bias." The poor certainly do not score as highly on average as wealthy students. Over the last forty years SAT scores have been positively correlated with family income...So the SAT appears biased against the poor in the sense that the poor tend to score lower and therefore will be less likely to be admitted to the college of their choice.

Sorry, but that's a definition of "differential impact," not bias. And while differential impact IS a topic that should be discussed thoughtfully, to lump this kind of effect in under bias is misleading. If the "poor" simply do not learn as much - quite possible given the likelihood of deprived homes and poorly-funded schools - then the test is actually performing correctly in assigning them lower scores.

Perhaps most surprisingly, there is evidence that these tests are biased against students with deep minds.

Pardon me while I snicker uncontrollably. "Deep minds" sounds like a concept you think about while passing a bong around. The SAT is test of basic skills which are very likely to come in handy for college classes. Will very smart - forgive me, "deep" - students find the test boring? Probably. Will it be less than useful for predicting how those students at the very high end of ability do in school? Most likely. But the only way that "deep" students will bomb the SAT in large numbers is if they fail to learn basic geometry and algebra, or how to discern the main point of a paragraph.

Anyway, that's my take. I also just have to point out that, although I started N2P as a way of rebutting anti-testing articles, I don't believe all test criticism is bad. There's plenty of room for debate on issues of standard setting, high stakes for young kids, differential impact as mentioned above, and so forth, and it would be great to see a website that discusses these issues without resorting to that old journalistic standby, "Critics say".

Unfortunately, Carter's site looks like it will just be rehashing the myths and bashing tests unconditionally, while not providing much of an alternative for states that want all their students to meet certain standards, or universities that are flooded with thousands of applicants each year. If testing is so bad, what's the best alternative? Carter believes it is, "Samples of work, references, statements of purpose, and extra-curricular activities," all of which are fine, but not necessarily verifiable, or comparable across students, or shown to be predictive of college success. Never fails to amaze me how people who will nitpick to three decimal places the predictive validity of the SAT will offer up, as an alternative, things like "statements of purpose" for which no predictive reliability data exist.

Oh, wait, before I go, I just have to quote this portion of his article, if only because I am amazed that he believes it is correct, or to the point:

Incidentally, despite having a mailing address in Princeton, New Jersey, ETS has no connection with Princeton University. Its luxurious headquarters, including tennis courts, a swimming pool and a private hotel, are in Lawrence Township, not Princeton. The Princeton mailing address is merely for public relations.

How EVIL! Actually, the Princeton link is very simple. In the 1920's, Carl C. Brigham, the Princeton professor who published A Study of American Intelligence, came up with his own version of the Army Intelligence exam to use as an admissions test for Princeton freshmen. Brigham was hired by the College Board (which is in NYC) to lead a committee to develop the test that eventually became the SAT, which was administered for the first time in 1926 - 22 years before ETS officially opened its doors.

The two founders of ETS, Henry Chauncey and James Conant, met Brigham in 1933 when they traveled to Princeton and decided the SAT could be of use for Harvard students, and the rest is history. Why they incorporated in Princeton rather than Boston, I'm not sure, but the first ETS office was in fact in Princeton proper, in Professor Brigham's original space on Nassau Street:

By the middle '50s, ETS was cranking out not only the SAT, but the Graduate Record Exam, Law School Admission Test and a host of Foreign Service and military exams - each exam swelling in importance with the size of the American educational establishment. The company was badly outgrowing its Princeton headquarters. Often, tests had to be prepared at the nearby firehouse on Chambers Street. When the fire alarm rang, employees had to push their work tables out of the way and make room for the engines.

The need to move was obvious. In 1954, Chauncey had a vision of its future when he took a hike along Stony Brook and saw a stretch of open farmland that looked perfect for ETS headquarters. ETS' move to its 360-acre campus in Lawrence Township was complete by 1958. It was a complex of low-rise, modern, brick buildings, but for the first few years it also shared space with a working dairy farm.

Therefore, the claim that ETS has no connection to Princeton University, or that ETS chose Princeton purely for PR, is both laughable and easily disproved through a bit of Googling. But I suppose the truth didn't fit with Carter's meta-theory about how all psychometricians are eeeevil capitalists, though.

Discovering Georgia's standards

The good news: After much dithering, the cutpoint for Georgia's CRCT reading exam for third-graders has been released (or, at the very least, leaked to the press).

The bad news: It's a pretty low cutpoint, which puts into perspective Georgia's triumphant announcement that 91% of their students passed.

To meet expectations, third-grade students this year needed to answer 17 of 40 questions correctly — just 42.5 percent. In Glynn County, 89 percent of students met that requirement on the reading test and will matriculate to fourth grade. The remaining 11 percent were targeted for summer school and took the test again Tuesday, according to a new state mandate that requires third-grade students to pass the reading test before moving on to fourth grade.

"What it's supposed to do is help us identify when students meet the minimum competencies on a test," he said. "That says you're on level. You're where you're supposed to be."

It's a minimum-competency exam. Fine. Now give us some measures of variability. What does the distribution of test scores look like? What's the standard deviation? How many students are within a few items of the cutpoint? Parents need this information to judge the claims of the state that 91% of its students are really ready to move forward.

Closing the gap in SC

Administrators at some South Carolina schools won't waste this summer at the beach; they're already planning the upcoming school year, with an eye towards closing the racial achievement gap, which continues to widen in the state:

It's summertime at South Elementary School. The halls are silent and empty. But behind closed doors, teachers, maintenance workers and school secretaries are busy, setting up classrooms and thinking about the coming year weeks before school begins in August...

Some of the report's recommendations include increasing instruction time for students who are at risk of falling behind, developing academic assistance plans for each child and improving programs for 4-year-olds to prepare them for school.

At some SC schools, students in the disadvantaged categories perform well on standardized tests, and other school hope to follow their leads. More on some of the successful schools here:

The biggest local winner was Stono Park Elementary in West Ashley, which has a school population that is 84 percent black.

She summed up her philosophy in two words: "We believe." Order and structure are also important, she said. "These seven (school) hours, we control. We don't wring our hands. We make use of it."

Shhh! It's a secret!

You say you want to know what the passing score is on Georgia's high-stakes reading exam for third-graders? Well, nyah-nyah, they're not going to tell you:

For weeks, the state Department of Education has refused to say how many questions Georgia's third-graders needed to answer correctly in order to pass a high-stakes reading test required for promotion to the fourth grade. Without knowing where the state set the standard, many observers say it is unclear what Georgia's surprisingly good performance on the test means.

State Superintendent of Schools Kathy Cox says the state did not manipulate the test to improve performance. The cut score — the number of correct answers a student must have in order to pass — was the same this year as in 2002, the last time the third-grade reading test was given, she said.

Yes, but while that tells us the standard wasn't lowered from 2002, that still doesn't tell us what it is. Keeping the cutscore in place doesn't guarantee that everyone actually improved, because the items could have gotten easier.

Standardized tests are not graded the same as classroom tests, which typically have a scale of 100 with a score of 70 being the minimum required to pass. Tests like the state's curriculum exam have determined cut scores based on how much a group of educators decides students need to know in order to prove they have mastered the subject.

On Gwinnett County's standardized Gateway exam, used for promotion in grades four and seven, a passing score is less than 50 percent of correct answers. The cut score is low because the questions are difficult, said Linda Mitchell, head of testing in Gwinnett.

It's quite possible that there's no reason for the hush-hush other than to satisfy the lawyers. I'm not going to jump on the conspiracy bandwagon. But knowing the placement of the cutscore does allow for more argument about how useful the test is, so naturally some have jumped to the conclusion that the DOE wants to forstall those conversations.

Say it's revealed that the cutscore is 40% of the items. This could mean the items were very difficult, or it could mean the cutscore is so low that almost anyone can pass. Since everyone's seen the test, the public has a shot at gauging the difficulty of the test, and can use the cutscore knowledge to get an estimate of what "proficient" really means. Texas reveals their cutscore, and it isn't any sort of infringement on copyrights or test security to do so.

Good news in Philly

On the heels of Philadelphia's crackdown on misbehaving students comes some good news - Philly's test scores are up:

The TerraNova test results, says school district CEO Paul Vallas, find that student scores continue to improve across the board:

"While we saw solid growth in our reading scores, the growth in math was very strong. And let me point out that the strongest growth was in the middle grades -- 6th, 7th, 8th, and 9th grades."

A biased article about "biased" items

This Times-Record (AR) article is about cultural biases on tests, but no evidence is provided to support the charges:

In remarks in Van Buren last week, President Bush said he is committed to narrowing an “achievement gap” demonstrated in standardized test scores among the nation’s students. To some extent, the gap is the legacy of decades of segregation, some area educators said...

Jim Hattabaugh, Mansfield superintendent, said his experience as a guidance counselor and administrator has shown him that most standardized tests have cultural and racial biases. Minorities tend to perform worse than whites on the test because the tests are authored by whites, he said.

And does he have proof of that? Does he know for a fact that each and every single item writer is white? Does he have any knowledge about the extensive item bias review that all standardized test items undergo? Is he aware that item writers, test developers, and psychometricians come in all sizes, shapes, sexes, and ethnicities? Does he have any evidence whatsoever to prove the causal relationship between the skin color of the item writer and performance of examinees of different colors on the exam?

I doubt it. He's saying this because he knows the reporter will not challenge him to provide proof.

“If you look at how an inner-city student is raised and what they’re exposed to, you ask those questions on a test and they don’t have a clue,” he said.

And why is the only legitimate explanation for that cultural bias? Couldn't it also be because those inner-city schools are not teaching children to read, they're failing to introducing them to concepts outside their narrow environment, they're not expanding their vocabularies, and they're teaching these kids that you have to "be white" to do well on tests?

“There could be some possibility of a cultural bias,” said Lavaca Superintendent Harvie Nichols. “I know, for example, when I look at textbooks, it’s difficult to understand, even with a college degree, what the textbook wants that child to do.”

What the heck does this mean? It could be that it's a bad textbook. It could be that Nichols college degree wasn't too useful. But why should we assume this anecdote proves that textbooks are culturally biased? Textbooks undergo even more review for bias than do test items. If Nichols can't make heads or tails of the textbook, then he should order new ones, instead of insisting that cultural bias must be to blame.

This is an example of shoddy reporting. The claims of cultural bias against tests in general - including the outrageous and inaccurate charge that only white people write test items - go unchallenged by the reporter, as does the unspoken assumption that inner-city students should never be expected to understand material that is not specific to their very narrow sphere of experience.

More FCAT opinions

St. Petersburg Times columnist Howard Troxler takes a look at all the recent FCAT numbers:

The most important news was that for the first time, more than half of Florida's kids in grades 3-10 are reading at or above their grade level. To be exact, it is 51 percent.

A naysayer might look at the same number and say: That's terrible! You mean that 49 percent of Florida's kids are NOT reading at their grade level? Indeed, as if on cue, Florida's Democrats issued a statement finding fault. "Victory!" sneered the Democrats' sarcastic headline. "Half of Florida Kids Can Read!!!"

Does this mean the Democrats agree that the FCAT Reading test does in fact test genuine reading skills? Hee hee.

Really, I thought Bush was fairly frank and realistic about the numbers he presented. For the most part they represent slow and incremental improvement...The best news in Bush's numbers came in the lower grades, where the most emphasis has been placed on reading.

In other words, a black kid in Florida is still only half as likely as a white kid to be reading at grade level. Find all the economic, cultural or educational excuses you want - this is a bedrock problem for Florida.

Do the Democrats have a solution to that, other than tossing the tests which spotlight the reading gap?

Round-up #2: Testing news

Still trying to get caught up on my "blogligations" here...

-----

Power to the people, man! Modesto High students protest standardized testing policies. You'd think I'd be mocking them, but the kids have their heads on straight. Their complaints - that the tests they must score "proficient" or above on are not in sync with their International Baccalaureate program - are pretty sensible. Their black t-shirts are pretty spiffy. And they're wearing these t-shirts as part of a silent protest. Teenagers who use common sense, wear black, and understand the power of silence - they've converted me to their cause already.

-----

High school GPAs are up, nationwide; test scores aren't. This editorial in the Las Vegas Review Journal wonders why.

-----

An Arizona State University survey reveals that parents like the idea of ranking schools with test scores and imposing exit exams. Ethnicity didn't matter, and the survey was conducted in both Spanish and English. It's a telephone survey, though, which means that people with only cell phones, or no phones, were left out.

-----

If you're an NYC third-grader and you missed the first reading test, the makeup is on May 12th. Be there, or be in third grade again next year.

-----

North Carolina is inching closer to dropping a test for teachers. But the Governor is fighting the change in plans, which was apparently pushed through to help battle a teacher shortage.

-----

Finally, in Louisiana's St. Charles parish, three teachers have flunked a different, very important kind of test:

...three teachers became cheaters themselves, betraying their profession, their school district and, worst of all, the children in their charge.

A teacher at Norco Elementary School gave answers to 15 fifth-graders who were taking the Iowa Test of Basic Skills last month, according to state education officials. A teacher at Ethel Schoeffner Elementary School in Destrehan looked at questions on the Iowa test and gave 15 sixth-graders examples of the types of questions they would be asked, officials said...

These teachers set a terrible example for their students. Ironically, it was the children who brought the incidents to light by telling another teacher what had happened. They showed courage and honesty in doing so, but they shouldn't have been put in that position.

Do as I say, not as I do

From this article on the STAR in California comes an odd, unquestioned testing criticism:

Sergio Miguel remembers all too well what it was like taking tests in a foreign language, something nearly 30 percent of Tulare County students do. Miguel moved to Lindsay from Mexico when he was a high school freshman.

"There is a huge frustration for me," he said. "If they were given a test in Spanish, without doubt their scores would be way ahead of their scores now."

Does Miguel not see that the reason he was able to succeed in our society was because Lindsay High School insisted on teaching him English? His students might very well do fine on the test if it were given in Spanish, but so what? Does that mean his students would have the tools necessary to succeed in our society? No.

That much said, I'm aware the tests aren't perfect, and testing does not in and of itself improve education. But I wonder if so many teachers would be "throwing their hands in the air" if there wasn't the constant, not-quite-accurate refrain about how these tests only measure affluence. Schools, and educators, who insist that accountability measures reflect only the home environment make me wonder why we should be paying their salaries.

More about the FCAT than you ever wanted to know

The big FCAT news: More than half (51%) of all FCAT-takers this past year are reading on grade level. Today's press conference webcast can be found here; score reports are on the same page. The reading scores suggest that schools in the lower grades are doing a much better job of educating readers than the upper grade schools; 70% of fourth-graders scored a 3 (out of 5) or better, while only 32% of ninth-graders did the same. There's no such pattern for math, though.

Third-graders with disabilities fail the FCAT at twice the rate as their non-disabled peers. 8,300 of them have flunked the test for the second year in a row, and more than half of those have a disability. There is a loophole: "special education students who already have been retained one year and have received intensive remediation are eligible to be promoted, even if they flunk the test a second time."

And speaking of loopholes: The 143 Collier County seniors who failed the FCAT but fulfilled all other graduation requirements will still be able to walk across the stage at graduation. The lone voice of concern ("Somehow our curriculum and instruction does not correlate with the FCAT. What should we be doing to prevent this discrepancy?") was bypassed in favor of making sure students don't become "another statistic in welfare." But if those students don't ultimately learn what they need to know to pass the FCAT, that graduation walk isn't going to get them hired.

Florida's senate said parents of failing students should be able to view FCAT items and answers; however, the measure, approved by the Florida senate three days before the end of this year's legislative session, did not make it through the House. The fight to open up the test to parents (and, on the part of education officials, to keep the test confidential) will resume next year.

More FCAT opinions and letters to the editor: The FCAT provides "a measure of reality". The FCAT is "not the solution". Retaining disabled students is somehow "breeding failure." And schools need to be teaching the skills measured by the FCAT, not the test-taking skills that have been the focus.

Last but not least, a bill approved by the state Legislature this spring allows seniors to substitute "passing" scores on the SAT and ACT for the FCAT requirement. The qualified students must have failed the FCAT three times but make a "passing" score on the SAT/ACT. In case you're wondering why I'm putting "passing" in double-quotes, it's because I'm skeptical about how such a standard will be chosen for a college admissions exam when the population in question has repeatedly failed a tenth-grade exam.

There's no reason to provide this alternative if the FCAT and SAT/ACT scores would identify the same group, so the thesis must be that there is some "passing" point on the college admission exams that students who fail the FCAT can achieve. But if the purpose of the exam is to "rescue" those repeated failers, the SAT/ACT standards will be set so low as to do nothing but verify that those students aren't too skilled.

Reading further, I see my worst fears are realized:

...the cutoff scores that seniors must earn to qualify for the substitution are a 15 each on the ACT reading and math tests. For the SAT, they must earn a 370 on the SAT reading test and a 410 on the SAT math test.

In other words, if you fail the FCAT three times, you don't get a standard diploma. But if you fail the FCAT three times and then, on the SAT, score around the 15th percentile on reading, and around the 28th percentile for math - scores which essentially show convergent validity with the failing FCAT scores - you can get a diploma.

To whom did this make any sense whatsoever?

To these state representatives, I assume:

State Rep. Joe Pickens, R-Palatka, said Thursday he proposed that the Legislature make the SAT or ACT tests permanent substitutes for the FCAT for seniors who fail the FCAT, but the state Senate did not agree. Pickens said that means the Legislature must approve a new law each year for the SAT and ACT rule to apply.

[Tony Hill Sr., D-Jacksonville] said in the release that "obtaining a high school diploma after diligently working toward it for 12 years should be joyous [sic] occasion, celebrating the end of the first phase of ones [sic] life and the beginning of the bath [sic] toward the remaining phases in ones [sic] life..."

If that's a verbatim quote, I worry about Senator Hill just as much as I worry about those students who learn so little from their Florida high schools.

Round-up: Testing news

The nation's class of 2006 have a choice to make - the new SAT, or the old SAT? The majority of schools will accept scores from either one, and students might feel their performance would be better on one of the two. And the two largest test-prep companies, Kaplan and The Princeton Review, are already sniping at one another about the best method of test preparation.

------

The subheadline for this story says it all: "District fears total collapse; voters tired of poor test scores." How poor? Considering that Southfield (MI) Public Schools is one of the most highly funded districts in Oakland County, pretty poor:

If the June millage passes, the owner of a $200,000 home in Southfield would pay $1,963.45 for schools in each of the next five years.

Despite the heavy tax burden, only 48 percent of all tested Southfield students passed the standardized Michigan Educational Assessment Program, or MEAP, test. That compares to 78.4 percent in Bloomfield Hills and 72.4 percent in West Bloomfield, according to Standard & Poor’s School Evaluation Services. The figure is a percentage of all students in grades four, five, seven, eight and 11 who meet or exceed state standards.

Some say the demographic specifics of Southfield mean that students just don't have opportunity. But with this tax crunch, pretty soon, there won't be any students left. A testing opponent is quoted as saying that Southfield's poor test scores aren't actually a reflection of ability, but instead of the amount of "opportunity" available to students. Why parents should pay so much money in taxes for schools that don't seem to be doing much to provide opportunity goes unexplained.

------

In Ontario they're disputing the 10th-grade literacy test because it "discourages" those who fail it. Presumably, those who who fail to attain a 10th-grade literacy level but are still awarded a high school diploma will not be "discouraged" by their subsequent life experiences. Apparently, there are tons of jobs in Canada that don't require tenth-grade reading skills. Who knew?

-----

They start young in Texas: Fifth-grade students to begin SAT test prep.

Too broke to test all three R's in NH

Cash-strapped New Hampshire will be giving fewer standardized tests this year:

State budget cutbacks mean New Hampshire students will take only two standardized assessment tests this year - reading and math.

Officials say they are doing the bare minimum to satisfy the federal No Child Left Behind Act. Lawmakers required the Department of Education to use federal No Child Left Behind money for this year's round of tests. When it fell short, tests were dropped.

Is the fact that New Hampshire changes the test from year to year part of the funding problem?

It's FCAT time!

FCAT scores, that is. The Miami Herald says Florida's third-graders did better than last year, perhaps as a result of the new mandatory retention policies:

Statewide, 66 percent of third-graders passed the reading FCAT, compared with 63 percent last year. In Miami-Dade, 57 percent passed, up from 53 percent; and in Broward, 65 percent passed, up from 63 percent.

''When we ended social promotion and raised standards for our high school seniors last year, many were skeptical,'' said Gov. Jeb Bush, according to a statement released by the Department of Education. "Today's results show Florida is moving in the right direction, with more students reading on grade level and significant improvement and opportunities among those who have struggled most.''

The Palm Beach Post also reports good news:

About 23 percent of St. Lucie County third-graders and 12 percent of Martin students are in danger of repeating third grade next year after failing the reading portion of the Florida Comprehensive Assessment Test, according to results released Monday.

The failure rate in both counties dropped from last year, when about 25 percent of St. Lucie students and 14 percent of Martin students failed the test. Statewide about 22 percent of third-graders failed the test.

As far as the 10th-grade FCAT goes,Governor Bush recently noted that 10% of Florida's seniors will face a barrier to their diploma based on FCAT scores. Let's find the press releases for ourselves on this. Ah, here we go. From here you can view the webcast from today or dig around for statewide and district-level results. The percent passing results from 12th grade alone aren't that informative, though, because that's just the kids who didn't take/pass the exam earlier.

Oooh, pie charts. Oh, wait, those are just telling us what students are doing after graduation (regardless of whether they passed the FCAT). Fifty-eight percent of those who failed the FCAT last year are enrolled in some kind of secondary education this year, presumably places that take a GED. The Florida DOE has a "Stay In The Loop" page for FCAT flunkers; if you know any senior who didn't pass the test this March, they might be interested in this information.

There's an interesting "FCAT Myths Vs. Facts" sheet available, too.

When young men's fancies lightly turn to thoughts of test items

Ah, the first signs of spring have arrived. No, I don't mean the 70+ temps, bright sunshine, hyacinths, or sandaled feet. I mean the parade of "SAT as a rite of passage" articles that are sometimes informative, and often critical.

This article, in the Tri-Valley Herald, suggests that SAT "mania" is gripping students like never before:

Today, however, more students than ever before are taking the SAT. The test has taken on near mythic proportions for high school students and their parents, who view a high score on the SAT as a magical Golden Ticket that, if it doesn't guarantee access to the most prestigious colleges, will at least boost a students' application to the top of the pile.

The College Board website note the increase in examinee numbers too, although they present this increase as a good thing, rather than something that should instill anxiety into students and educators. For example, some 38% of all SAT-takers are the first generation in their family to plan for college. But, as the TVH notes, budgets aren't keeping up with the "bubble" of college hopefuls:

A massive bubble of kids are graduating from high school and pushing at the doors of the nation's colleges and universities. Competition for slots is fierce, and students, aware of the pressures, are sending applications far and wide, thus adding even more competition for already scarce slots.

In deficit-ridden California, the storm is worsened by budget cuts that are, for the first time, forcing the University of California and California State University systems to turn eligible students away. By some estimates, as many as 23,000 California freshmen who in better times would be welcomed at CSU and UC campuses won't find a slot in the fall.

The article is honest enough to note that, despite some public anxiety, college admissions are not all about SAT scores:

The truth of the matter, however, is that many colleges don't weigh SAT scores as highly as they once did. In fact, the use of SAT scores in admissions at UC has been a hot topic this year, sparked by a critical report from the chairman of UC's governing Board of Regents that slammed UC Berkeley, the system's flagship campus, for admitting 386 students who scored 1000 or below on the test and turning away 3,200 who scored 1400 or above...

SAT scores are a factor in UC's admissions decisions, but they don't carry as much weight as other factors, including a student's overall grade point average and his or her scores on the SAT IIs, the subject-specific standardized tests that students must also take in order to be considered for UC admission.

The Philadelphia Inquirer, on the other hand, is all about the pressure of the SAT:

Although the new test is nearly a year away, it's been the topic of discussion and planning among colleges and high schools, tutors and guidance counselors - as well as among some students and parents.

"The poor kids in the first class to take it will be nervous," said Carol Lunkenheimer, dean of undergraduate admissions at Northwestern University. "They'll be saying, 'Woe is me - why is my class the one taking this exam?' "

What would we do without these critics who have convinced nervous students that the College Board cares only about money? Surely, these naysayers play a large part in creating examinee anxiety with their accusations, but the Philadelphia Inquirer article (which also includes the classic, throwaway, unsupported statement that "critics have long held that the SAT contains class and cultural biases that hamper minority, poor and working-class students") would rather ignore that little detail. Funny, too, how the PI doesn't note that the same critics who are screaming about market shares following the SAT revisions would have been the first to scream about obsolescence and stubbornness if the College Board hadn't modified the test in response to criticism.

Meanwhile, the Orlando Sentinel noticed something escaped many other papers. While more students are aiming for college, fewer of them are willing to tell the College Board their race:

Last August, as it does every summer, the College Board released its national SAT report, showing that the average score on the college entrance exam had climbed slightly, as had the scores of minority students...buried in the data was a fact overlooked by researchers and journalists: A record portion of the test-takers, 25 percent, had declined to disclose their ethnicity.

That's triple what it was seven years ago. And there's no reason to assume that students who refuse to disclose their race can be left out of analyses, or that such students are similar to those who do list their race. There is research (by Dr. Dale Whittington and Dr. Howard Wainer, both of whom are quoted in the article) to suggest that such omissions undercut the ethnic-analysis conclusions that have been drawn from SAT scores in the past.

Some claim that the increase is due to students trying to avoid "stereotype threat" or AA policies. The College Board claims instead that the previous registration forms were too accommodating, and have noted a sharp drop in ethnic-group nonresponders now that students are forced to choose a race category (including, "I choose not to respond").

Skipping spring break

I doubt these third-graders in Savannah (GA) had planned to spend their Spring Break in Daytona Beach, but I'm sure they'd rather be somewhere besides the classroom:

About 1,100 third-graders who need extra help to pass the reading portion of a state standardized exam in a little more than a week, were invited to give up a traditional spring break in exchange for a more educational one. Third-graders who don't pass the reading portion won't advance to fourth grade, according to a state law now in effect. So being prepared to pass is crucial.

Roughly 75 percent, or 854 kids, took the school system up on the intensive Third-Grade Reading Camp offer, and have been attending school for three hours each day this week. Educators say they believe results will be positive – more kids will pass the exam.

The test they're preparing for is indeed the same CRCT that James Hope is so unhappy with (in the previous post, below). This article provides links to sample items, none of which seem as strange as the one Hope cites.

I particularly like #5, too, because the most recent lead singer of my boyfriend's band was named "Goat."

Hope lives on in Georgia

Gwinnett County (GA) teacher James Hope is NOT happy with the state of testing in Georgia right now:

Here's the question: What change should be made to the phrase "stir it around" in the sentence below?

That's nine redundant tests in one month.

Hope then goes on to mention that he got in trouble for posting Gateway test items online before. I thought I recognized his name.

I would love to give you a specific example of a Gwinnett Gateway test question, but the last time I made some of those questions public, in an attempt to show how some test items do not measure what is taught in schools, I was visited three times by school system police, had my phone records confiscated and almost had my teaching certificate revoked.

Almost - as I reported previously, Hope was cleared of wrongdoing. I don't blame him for questioning items of the type above, and I agree that Gateway's confidentiality agreement is pretty harsh (for most tests, only live items are covered under such agreements; Hope posted sample items). And I agree with him completely on this:

These tests were supposed to be the yardstick to measure educational reform, not be the educational reform. And that's true, no matter how you "stir it" or "stir it round and round."

Testing as a worthy sacrifice

Students entering graduate programs say the admissions exams are worth the worry, because it gives them a taste of what the workload will be like:

Looking toward the future, after the endless test practice, essay writing and anxiety of waiting can often be difficult. But for students who have climbed the mountain and now find themselves in UCLA graduate programs, many see the sacrifices made in preparing applications for graduate school as entirely worthwhile.

The LSAT tests the processes of logical thinking that are important to completing law school. "I thought, 'Wow, I'm enjoying the LSAT – I'll probably enjoy law school,'" she said.

Bingo. It always amazes me when students planning graduate careers complain bitterly about the MCAT, the LSAT, and the GRE. They don't like having to work quickly, read a lot of material, synthesize a lot of information, and answer items that are based on rigid standards? What on earth do they think graduate school is going to be like?

"The grand exercise in wishful thinking" about test scores

Professor Amy Wax, in an absolutely phenomenal WSJ Op-Ed, says that the answers to solving test score gaps are not easy, despite recent research suggesting that minority students falter mainly because of deep-rooted stereotypes. After noting that the existing "stereotype threat" research may be fundamentally flawed, Professor Wax lists the incorrect yet popular anti-testing assumptions that are at the root of much educational research today:

Lack of evidence and grave methodological defects haven't prevented the stereotype threat industry from taking off. Distortions are now pervasive. According to a survey by Mr. Sackett and his colleagues, 10 of 11 references to the threat in scientific journals, more than half the descriptions in psychology textbooks and 14 of 16 discussions in the media incorrectly state that racial differences in academic performance disappear when stereotype threat is eliminated. In this vein, a recent New York Times article on stereotype threat and the racial test-score gap declared that "performance is psychological." A Frontline special falsely stated that blacks who believe that a standardized test was merely a research tool, rather than a gauge of their abilities, performed just as well as whites.

• The first item of faith is that any assessment that reveals group differences must be biased, inaccurate and invalid. If scores can be lifted merely by adjusting attitudes or test conditions, then poor scores don't reflect real deficiencies in knowledge and ability and tests aren't an accurate measure of academic skill.

• The second notion is that groups don't really differ in academic proficiency or learning. Stereotype threat is a temporary brain freeze that covers up what students really know. If performance across groups can be equalized just by dispelling the test-takers' fear of being judged, then current disparities reflect no real group differences in learning or skill.

• Third, stereotype threat promises a quick fix for low achievement. We resist the idea that high test scores reflect dedicated study and good learning habits, that learning builds on itself over time, and that there may come a point when past deficits can't be made up. We want to believe that anyone can always catch up and that latent potential can be instantly unleashed if only the right formula is found.

• Finally, we resist confronting the social and behavioral causes of shortfalls in academic performance. Stark differences between groups in marriage rates, family stability, paternal involvement, parenting practices and discipline, and other habits and values, are associated with children's disparate academic success. Changing these requires sustained self-scrutiny and reform from within. We'd rather believe that underachievement comes from without. If only white society would stop stereotyping minority students as inferior, or expecting them to perform poorly, stereotype threat would abate and all would be well.

Emphasis mine. I bow to Professor Wax's thorough yet concise description. There's no better way to sum up the hysterical reaction to testing, and to the ethnically-charged achievement gap, that one sees almost everywhere these days. "A grand exercise in wishful thinking" - I'm going to have to use that phrase more often.

This is not to say that there are no legitimate criticisms of testing. Tests can be used improperly. Testing content can be inappropriate. Tests can be given too often, and the stakes can be set too high or too low. But every reporter who covers an exit exam or admissions exam controversy who unthinkingly prints the line, "Critics say such tests are biased...," should be forced to read Professor Wax's explanation of why bogus test bias theories abound, and should be forced to explain why they feel it's unnecessary to let their readers in on the reams of research demonstrating that a very real achievement gap underlies the legendary test score gap.

Her summary?

People who don't know how to read and do math can't function and lead in a demanding, technological society, regardless of the cause of those shortcomings. The insight that anxiety about stereotypes may influence minority students' real learning is not without implications. It suggests that encouragement and reassurance are a vital part of education. But in urging students to prove their detractors wrong, one key message should never get lost: What matters most in the end is what you know.

And in the end, if you know it, you can show it on a test, no matter what group you belong to.

(Thanks to Devoted Reader EGF for the link.)

Testing information for parents in the Grand Canyon state

The Arizona Republic supplies a handy summary of testing information called, "Which Tests Matter?"

Arizona students take two types of standardized tests.

• Before they were hired, teachers once had to stand on a stage where they were grilled to make sure they knew the answers to standardized questions.

Substituting the SAT

Teachers in Virginia can now use their SAT scores in place of the Praxis I. The hopes is that it will simplify the process and perhaps encourage those who might not otherwise consider teaching to enter the field:

If teacher-candidates took the SAT after April 1995, they would need to have earned at least 530 out of a possible 800 on each section, mathematics and verbal, and a combined score of at least 1100. For those who took the SAT before then, a minimum of 450 on the verbal section and 510 on the math section is required, along with a combined score of 1000. The differences reflect the "recentering" of the SAT...

But, he added, the board does not expect a lot of new teachers to fall into that category.

I'll say. As ReformK12 points out, high school seniors planning to major in education have typically had a very low mean on the SAT. In 2003, that mean was around 965 for a combined SAT, which means that (if one makes some assumptions about the shape of the distribution), only around 16% or so will have a high enough SAT score to opt out of the PRAXIS. Changing the test isn't lowering the standards, in this case; I think it's debatable whether this change will encourage more high performers to enter the teaching field.

In an ISAT state of mind

The Champaign, IL News-Gazette has a long article on the new rites of spring, the ISAT (Illinois Standards Achievement Tests) (thanks to Devoted Reader Michael S for the link):

Cheryl O'Leary, principal of Champaign's Garden Hills School, and her teachers are getting students there psyched up for the ISATs to be given this week by planning a series of special events at the school and by giving them incentives to show up on test days.

"We're on countdown for the ISATs," said Cheryl O'Leary, principal of Garden Hills School last week of pretest and test-time activities that included a Wheel of Wisdom – like Wheel of Fortune – presentation Friday sponsored by the PTA to get students excited about taking tests.

That seems like it would be as difficult as turning lead into gold. I appreciate O'Leary's enthusiasm, but do kids need to be made excited about everything in school?

One director is refreshingly pro-testing:

"High stakes tests like the ISAT have influenced districts greatly because of the accountability factor," said Mary Muller, the district's director of elementary curriculum...She said administrators take a positive approach to testing, which is sometimes criticized for taking up increasing teaching time in classrooms and for forcing educators to tailor their teaching to what's asked on the tests.

"You hear the negatives, but these tests really can give everyone a better picture of how students are doing so we can improve instruction," Muller said. "That's the missing piece. The ultimate goal is to raise student achievement, and the way to do that is to align with state standards and to assess what we're teaching."

Stephen Lucas, principal of Edison Middle School in Champaign, gets it as well:

"As far as this teaching-to-the-test thing goes, my idea is that if it's a good test that really measures state standards and what the district wants to achieve, of course it's a good thing," he said. "If you have good standards and the tests measure them, of course we want to teach to the test."

Actually, it's my hunch that a lot of teachers, principals, and administrators are pro-testing and think the same way as Muller and Lucas. The news media, however, tend to take a relentlessly anti-testing stance, and it's rare to see a news article like this one prominently feature pro-testing comments.

Catching them at an early age

I love this intense, yet enthusiastic first-hand description of the gifted-and-talented test being administered to kindergarteners in the Kerrville (TX) Independent School District:

It was kindergartner Dillon Young’s first time with a standardized test, and it was hard.

As for the questions themselves? “Oh, they were fun,” Dillon said with a small grin.

The assessment of children this young is new for the Kerrville G&T program, and the test results will be considered in conjunction with teacher reviews:

Anybody, including teachers and parents, can nominate a child for the program. Then those students are given different tests to determine their aptitude and ability.

“I want to learn how to times (multiply), and how to read,” he said with a big smile.

NCLB as scapegoat

Here's an article that is sympathetic to the problem of "overtesting" in schools, told primarily from one elementary school teacher's viewpoint. The article is straightforward and fairly informative. But can you spot the simple solution to this problem?

Under President Bush's "No Child Left Behind'' bill, [this teacher's] district received a three-year grant worth nearly $5 million to implement a literacy program in kindergarten-through-third-grade classes. Similar grants totaling $900 million went to thousands of low-income, low-performing schools across the country.

I know a teacher who would like that blueprint. The one she has now isn't working so well.

According to this teacher, the school was already implementing a successful literacy program, one which apparently conflicts with "Reading First." But why is the blame being placed on NCLB here? The blame should be on the school for taking the $5 million without understanding the accountability that comes with such a large chunk of money. It's not surprising that additional testing comes with additional money, but neither the money nor the additional testing is mandatory.

What conclusion are we supposed to draw here? That NCLB is bad because schools are required to account for grant money spent? That it's unfair when some reading programs conflict with one another? That students who are in federally-mandated reading programs shouldn't have to follow federal rules? This article is, as I said, sympathetic, but this teacher is barking up the wrong tree.

High prices and suspicious claims

From the Seattle-Times, a peek behind the curtain of the test prep industry:

Just as aspiring actors must try out for parts, aspiring test-preparation teachers have to audition for their roles.

They hold regular auditions — but only for those who've scored at or above the 90th or 95th percentile on such tests as the SAT; Graduate Record Examination or GRE; and the Medical College Admission Test, or MCAT.

Such programs aren't cheap, despite the fact that few studies exist to show a clear gain in test scores after enrolling in such programs. The test prep instructors also make a big deal out of being able to quell test anxiety. However, I've always felt that, with their high prices and their public insistence that the tests don't actually measure anything yet are too tricky for examinees to prepare for on their own, these test prep companies are part of the test anxiety problem.

You don't need to spend a thousand bucks to pass these exams, especially if you're already smart, disciplined, or accomplished (because the tests do measure aptitude and ability). If you're really not smart, or you haven't paid attention in school at all, learning the "tricks" won't help you anywhere near as much as the test prep companies claim.

The unbearable requirement of learning to read

The issue of the MCAS (Massachusetts Comprehensive Assessment System) is "as divisive as any issue facing schools," according to the Boston Globe. But not for the reasons you might guess. The Globe saw fit to print quotes from a testing critic who does nothing but rehash the extreme-PC objections to standardized tests:

Marcella Lang, an elementary school teacher since 1975 who now teaches English as a second language in Somerville, wants the MCAS abolished...

Marcella: I think of MCAS as a classist, racist test. It is unfair, and it contributes to leaving behind the very students it claims it wants to help. It doesn't take a genius to see that children from affluent communities and with educated parents do very well on the test. And who doesn't do well? Special-education kids. Trade-school kids. Minorities and underprivileged kids.

And does Marcella believe this bias exists because of how the test is constructed, or because the items themselves are biased? In other words, does she have any substantive commentary on the quality of the test?

Nope, the MCAS is "classist and racist" simply because it's unfair to expect all children to perform well:

[Children are failing the MCAS because] the exam is worded in a way that loses a lot of kids and is designed in a way that is very difficult for a lot of kids to process. It's unfair to expect the same from kids who have been read to since they were born and children who have never seen a book, never been in a library.

Emphasis mine. It's unfair to expect kids who come from deprived backgrounds to, you know, learn anything in school (as Best of the Web put it, isn't that what schools are for?). The test is "classist" because kids who were read to at home aren't just lucky; they had an unfair advantage, and presumably should not be allowed to reap the benefits of high test scores. Why, what have all these parents who read to their kids been thinking? Don't they know how unfair their homes are?

The amusing part is that Marcella's daughter, Marina, supports the test. The irritating part is that Marcella is given both the first and the last words in the Globe article, and her daughter's responses, unsurprisingly, were much more polite and restrained than mine.

Relaxation techniques for kids

One elementary school's response to test anxiety - playing classical music and letting kids chew gum:

Students at Hazelwood Elementary were allowed to chew gum and listen to classical music as they started the Tennessee Comprehensive Assessment Program (TCAP) Monday. Given statewide, the test is used to measure how students in third through eighth grades perform compared to others in the state and to students nationwide.

"We took advantage of every accommodation the State Department of Education allowed us," Hazelwood Principal Rhonda Kennedy said about students chewing gum. "Some research shows it's actually a tool to help them stay attentive."

The Tennessee DOE does indeed have a page on how teachers can help alleviate test anxiety, and a page on allowable accommodations; I didn't see anything in there about chewing gum, though. Wonder if blowing bubbles would be out of line?

More information on reducing text anxiety in children can be found at the US Dept of Education site and the United Federation of Teachers site. Interestingly, the UFT site's suggestions are all discussions between parent and child, with the "most important" tip being to remind your kids that "you will love them no matter what happens." The US DOE site, on the other hand, suggests more non-emotional action on the part of parents, from providing a good breakfast and quiet study time to making sure the kid attends school regularly.

Providing inspiration without information

Philadelphia students, if you're having a problem with standardized tests, you're not alone:

Half of the district's middle school teachers who took tests to become certified as highly qualified under the federal No Child Left Behind law failed, district results show. Math teachers did the worst: Nearly two out of every three failed that exam, while more than half flunked the science test, 43 percent the English exam, and 34 percent the social-studies test.

Philadelphia teachers failed the test at a far greater rate than those in the rest of the state...

Allegedly, this is no "wimp test," and alternate criteria for certification are being considered, but it's difficult to understand how Philly's teacher could be effective if they don't have mastery of the content material. What's more, the particular certification rules in Pennsylvania mean that elementary-certified teachers can teach middle-school classes:

In Pennsylvania, elementary teachers are certified through sixth grade and secondary teachers from seventh through 12th grades. But for schools that span both elementary and secondary grades - middle schools - the state has allowed elementary-certified teachers to teach all grades.

Most middle school principals in Philadelphia have preferred elementary-certified teachers to maximize scheduling flexibility; for instance, a math-certified teacher could teach only math, while elementary-certified teachers could teach all subjects. More than 90 percent of Philadelphia middle school teachers are elementary certified.

Needless to say, certain teachers think the certification tests are "not fair:"

The Philadelphia Federation of Teachers says it is pleased that the district plans to offer a test-preparation program for teachers, but questions the fairness of the testing requirement. "We have so many middle school teachers who have been doing a terrific job all along. They've been doing it for years," said Arlene Kempin, chief personnel officer for the teachers' union.

Doing it for years doesn't mean they've been doing it well. The Philadelphia test scores would suggest that middle-school teachers haven't been doing a bang-up job.

...Nick Perry, a science teacher at Conwell Middle School, said one test was not an accurate measure of a teacher.

"Content sometimes is really overrated. A teacher is like an artist, a coach. He has to be able to inspire children," said Perry, a seventh-grade science teacher, who has a master's degree in environmental science and the necessary certification.

Isn't that who you want teaching your kids - someone who thinks that content is "overrated?" And just how inspirational can a coach be if he doesn't know the rules of the game?

Testing is not the enemy

Samuel J. Green Middle School (LA) is a failing school that may be taken over by the state if students perform badly on the LEAP and Iowa tests this week. Now the principal, Paulette Walker, is also in hot water for passing out prayers with the test booklets that call on God to help kids defeat the enemy - i.e., the tests:

School officials will decide whether to reprimand Principal Paulette Walker after investigating her distribution of the prayer this year and another prayer notice to teachers last year, Assistant Superintendent Matt George said Thursday...He supervises the district which includes Samuel J. Green Middle School, where the prayer was given out Monday with test booklets for two standardized tests.

The prayer, which had errors in grammar and punctuation, states: "I receive your help faith, knowing that through you I shall do valiantly, for you are the one who treads down my enemies.(LEAP, Iowa)".

Emphasis mine. Teachers at Green got a "special announcement" last year from Walker that was similarly error-ridden, and which stated that, "Education in New Orleans, is in trouble." I'll say. I can't decide which is worse - the statement that testing is the enemy, the principal's attitude that prayer was necessary for her students to pass tests, or the principal being unable to convey her religious thoughts without grammatical errors.

Where algebra fits into the economic equation

Remember my previous post about the requests to remove the algebra requirement from the California high school exit exam? Lance Izumi accuses Californian schools of dumbing down the tests and most likely depriving their high school graduates of future computer-based jobs:

In communist China, computer software colleges are being built at 35 universities around the country...Within three years, the software college at Peking University will have 3,800 students specializing in subjects such as integrated-circuit design and information security. Much of the instruction will be in English. In China, 58 percent of the degrees awarded in 2002 were in the physical sciences and engineering, compared to just 17 percent in the U.S...

And our response to China's threat to overtake us in technology development?

So how is California, home of Silicon Valley, meeting this foreign challenge? The latest trend has been for school districts to plead with the state to waive the algebra requirement for high-school seniors to graduate this year. Judy Pinegar, manager of waivers at the state Department of Education, says that the number of districts asking the state for waivers “is increasing algebraically” and that the Department is “getting tons of calls.” State lawmakers will likely introduce legislation to postpone the algebra requirement for at least one year.

Isn't it rude of Ms. Pinegar to use the term "algebraically" when so many California high schools have admitted that they cannot teach those concepts within four years? Isn't it a given that many of their students won't understand that term?

Mr. Uzumi's conclusion?

The global economic race will be won in part by the quality of education of countries’ workforces. Too many of our educators whine about diverse student populations and racially biased tests, while our foreign competitors focus on high expectations and merit. If our educators fail to see the bigger economic picture, they are consigning our nation to a very scary future.

Protesting the PSSA

A minor student revolt in Pennsylvania:

Mennies was one of 25 students who traveled to Harrisburg Wednesday to lobby state legislators about the need to de-emphasize the importance of standardized tests. The group of students, known as Students for Legislative Accountability...included students of various academic achievement levels from each high school grade.

The students, who met with 10 legislators or members of their staffs, were well-prepared. They offered the lawmakers their voices in face-to-face meetings. They presented them with a packet of information that included written personal opinions, statistics and various newspaper articles about their concerns.

And we all know how balanced newspaper articles are when it come to testing, right? This article is typical; I noted that one student quoted has dyslexia, another student said that the 3 to 7 days needed for testing makes a huge impact on students, and yet another complained that tests are just a "glimpse" of learning. True, but the best possible glimpse for the purpose.

One students complained about the effect that tests have on the curriculum, but if their teachers can't teach basic math without teaching only what's on the test, isn't there a problem with the quality of teaching as well? These students sound genuinely concerned that their curriculum has been dumbed down, but the test doesn't require that.

Out of the frying pan, into the fire?

One of the problems with tests that assess students at the state-, school- or district-level only is that if individual scores are low-stakes for the students, motivation is low. But this is not the solution to that problem:

Kentucky students might soon find their statewide test scores on their school transcripts — there for everyone from colleges to potential employers to see.

In the past the panel has said the scores shouldn't be included on transcripts because subjects are not tested in each grade level, making it difficult to get a complete picture of each student's performance, state officials said.

At the very least, students who took the CATS under low-stakes rules shouldn't now see those scores on their transcripts. Parts of the CATS include writing portfolios graded by school and district personnel, which is pretty subjective and unlikely to be reliable at the student level. And if colleges or employers are unfamiliar with the test and there's no proof that CATS predicts either college or job performance, the scores are useless to them.

Dumbing down the tests

The Sacramento Bee notes that, due to "confusion among some school districts," the algebra graduation requirement will be waived for California's high school student this year:

Thursday's decision by the state board was welcomed by some educators and attacked by others, including students, who said it sends the wrong message to young people who worked hard to meet the requirement.

"High schools have known for four years that the class of 2004 had to pass an algebra proficiency test or algebra altogether," she said. "The administration should have been looking out for them. And students knew they had to buckle down."

Yes, they did. Despite all the concern shown in this article for students "caught in the middle," bear in mind that those are students who completed four years of high school without ever managing to finish this basic course. Did no one notice that these students weren't moving forward in math at all?

Some districts said they were unaware of the Algebra 1 requirement even though state officials had notified local districts more than once. Others said that when the state postponed the exit exam requirement, they thought algebra also was included.

So this cluelessness is being "rewarded," by pushing kids out of high school with a diploma that will be pretty useless if they plan to pursue higher education or hold a job that requires understanding of math above the eighth-grade level. Way to go, districts.

Even districts that knew the score plan to apply for waivers, presumably because their algebra courses are ineffective enough to require two to three go-rounds:

In San Juan Unified, 4,000 seniors already have passed Algebra 1. Nearly all the seniors currently enrolled in the class are taking the course for the second or third time, school officials said.

Currently, 629 San Juan seniors in 14 comprehensive, continuation and charter schools are enrolled in Algebra 1 or 1B to meet the requirement. Of those seniors, about 150 are special education students.

In other words, 76% of those who didn't manage to master algebra in four years are not special education students. There's no rationale for giving them a pass; the San Juan director's comments about the lack of "extensive tutoring" is ridiculous. And why is everyone bending over backwards to be "fair" to the 13% of San Juan seniors who haven't passed the course, when the other 87% did their work? Is it "fair" to those who succeeded to give everyone the same diploma?

Many educators and students have stressed that most students can pass Algebra 1. "I don't understand people who are still taking algebra their senior year," Sac High's Anderson said. "They have had four years to pass it. That is four years to understand the concept of balancing equations and solving for X."

And the legislators are already trying to get involved, in hopes of making things worse:

Meanwhile, two state legislators are considering introducing legislation to postpone the algebra graduation requirement for at least one year. Assemblyman Darrell Steinberg, D-Sacramento, said he wants to make sure that no student is deprived of a diploma without ample notification of the algebra standard and adequate opportunity to pass the class. He said he has been discussing the matter with Sen. Charles Poochigian, R-Fresno.

Yeah, we can't have students who don't understand high-school math being "deprived" of that piece of paper that will be of such service to them. This confusion of the diploma itself with the skills behind it - as though the diploma, and not the coursework, confers understanding and mastery - never fails to astonish me.

Back on the East Coast, a Florida legislator wants to know if the FCAT was dumbed down:

Senate Minority Leader Ron Klein of Boca Raton said he has fielded many calls from teachers who believe this year's FCAT is considerably easier than last year's when thousands of third-graders failed and were retained.

States contract with testing companies, which create questions that are tested to determine their validity. Tests use new and repeat questions, so that they have a similar degree of difficulty each year, said Jay Greene of the Manhattan Institute, which has studied standardized testing.

Anyone got any inside information on this? Certainly, a heavy reliance on repeat items jeopardizes test security; copies of old tests can be used to study for the new test, and that sort of cheating would explain the testing behavior.

Doing his part in the defense of tests

Richard Phelps, an acquaintance of mine who was very supportive when I began this blog, has finally brought his book, Kill the Messenger: The War on Standardized Testing, to fruition. It's been in the works for a few years, and I'm very happy to see that it's now out there for the general public.

Here's a review from Enter Stage Right:

A week doesn't go by, without a mainstream media story on the "horrors" of standardized testing, in which reporters tell of widespread testing error, of how testing is causing students to drop out of school, or of how testing is causing an epidemic of cheating.

Or how testing prevents teachers from actually teaching, or prevents students from actually learning, or from having any fun...but I digress.

The story behind the stories is that the relative prevalence of testing error is infinitesimal, that journalists stressing the dropout factor are mindlessly repeating a myth invented by radical Boston College teacher education professor Walter Haney, and that cheating is more easily prevented on standardized tests than with their alternatives.

For years, the American public has been force-fed a diet of test-bashing by the establishment media, the teachers' unions, professors of teacher education and well-financed anti-testing organizations, in which test-bashers have twisted existing data, ignored contrary data, and fabricated data outright. So reports Richard Phelps in his brilliant, new book, Kill the Messenger: The War on Standardized Testing...

Phelps argues persuasively that objective, external, standardized, high-stakes testing is the best measure we have of how much students have learned, and how well teachers, curricula, and textbooks have done their respective jobs. The tests give us a tremendous amount of information on children's academic strengths and weaknesses, so that we may help them improve. "Objective" is in contrast to classroom grades, which are increasingly subjective, politicized, and inflated. "External" means that school officials with a stake in the results do not control examination grading. "Standardized" means that a test "is given in identical form and at the same time to students in more than one school, and all the results are marked in the same way." And "high stakes" means that test scores have consequences, so that the test serves as a powerful motivational tool. Alternatives such as classroom grades and "portfolios" of work lack the advantages of standardized testing, while being much more vulnerable to manipulation and cheating.

Go forth and purchase. I've already ordered my copy. The book's already gotten 5 reviews on Amazon. Don't miss the comments from the reviewer who gave it only 1 star; she says virtually nothing about the book, choosing instead to relate her personal tale of woe because she didn't receive accommodated tests from some Texas universities:

Confusing rote obedience with intelligence, the authors selectively ignore cases (I and many others) that could not pass our state's standardized exams (now the political vogue) yet maintain a 4.0 average, ironically the mark of excellence. This is not an accident or misprint, but reflects a calculated war against anybody labeled different...

Unlike components for the general degree plan, the 'accommodations' option (regardless of how simple the provision such as a four function calculator, colored overlays etc...) for Texas's higher education testing program is not available at every state institution, wrongfully implying that disability is an 'extra', and reinforcing the idea students with disabilities are not 'real' members of the academic community. Once we are devalued, it is easier to justify overall discrimination against people with disabilities.

What does this have to do with Phelps' book, again?

Update: An archived version of Linda Seebach's review of the book for the Rocky Mountain Times can be found here. She neatly summarizes the motive behind the anti-testing bias so often seen in the media:

The unspoken difficulty with the SAT is not educational, it is political: namely, Asians and whites consistently score on average a couple of hundred points higher than blacks and Hispanics. And it isn't because the tests are biased, any more than scales are biased because they consistently show that men, on average, are heavier than women. Tests predict almost equally well for all races.

Nobody wants testing to reveal these differences, but nobody has any idea how to change them, either, so the only way out is to look for other excuses to lessen the importance of the SAT and other similar tests.

Fighting the test in Colorado

A retired Colorado teacher is spending his time trying to "defeat" the state standardized test (warning: annoying subscription process required):

If Don Perl gets his wish, the Colorado Student Assessment Program tests won't exist anymore. Perl, an opponent of CSAP, wants to kill the standardized test through the ballot box.

"The threat is that if those schools don't do well, they will be taken over by the government and turned into a charter school," Perl said. "Who knows what chaos that will bring."

So Perl's only opposition to the tests is that if the scores indicate schools aren't doing well, the public system might be forced to give parents a choice? Why is "chaos" the only option that Perl can see?

The state uses the CSAP scores as part of the School Accountability Reports. If a school receives an unsatisfactory score three consecutive years on the reports, the state can turn that school into a charter.

Interesting. I wasn't aware that this was part of the NCLB; this must be Colorado's decision. Given that the tests are required as part of the federal act, though, I think it's unlikely the tests will be removed even if voters come out against the CSAP. Information on the CSAP can be found here. Colorado had pre-2001 laws on the books requiring the testing of kids from kindergartern to 10th-grade, but there's no indication that, before NCLB, schools doing poorly would be taken over.

And here's at least one person who will sign Perl's petition, if she hasn't done so already - Angela Engel, guest commentator for the Denver Post:

...Under the guise of accountability, policymakers have made test scores the complete indicator of what children will learn in Colorado. Instead of questioning that reasoning or examining the research around standardized tests, the public has blindly followed...

Um, it's not so much that the public doesn't "question" the research supporting standardized tests (which is substantial) as the public doesn't often see very solid information about the pros and cons of using these types of tests for this purpose. Even when journalists and commentators have operated with a solidly anti-test bias, rarely have they provided the solid research that would allow their readers to question the testing policy decisions.

Test scores have become the Holy Grail in education. By the time Colorado students graduate from high school, they will have spent, on average, a full 52 weeks being tested. The National Center for Fair and Open Testing reports that students will take 36 to 60 standardized tests during their K-12 careers.

"National Center for Fair and Open Testing" - yep, there's an unbiased opinion. As for 52 weeks being tested, I'm skeptical. Here's a breakdown, assuming the following - schools meet for 275 days per year for 7 hours per day:

1st through 12th grade = 12 (275) = 3300 days total, at 7 hours each = 23100 hours total for education

52 weeks = 260 days (five per week) = 1820 hours (assuming 7 hours count as a day)

1820/23100= 7.8% of total hours spent in school, or a little less than 1/12th.

For a 180-day school year, which is probably more realistic, this works out to be around 12%.

Assuming equal distribution of testing time per year (at 1925 hours per year for the 275-day model, and 1260 hours per year for the 180-day model), we arrive at a figure of between 21 and 22 days of standardized testing time per year (regardless of model). But according to this information, the four CSAPs are administered in each grade once a year, and testing sessions last only three hours each. That's 12 hours, or less than two days - where are the other 19 days of testing? Even with the ACT/SAT and NAEP in there, I don't see how this could be.

(Update: Kudos to Devoted Reader Zach, who noticed that the article doesn't say students spend 52 weeks taking standardized tests, but just tests. By following it immediately with the alarmist statement from FairTest, someone who's not reading closely - like me - doesn't notice that the qualifier "standardized" isn't there for the "52 weeks" number.

If the 52 weeks include every type of test, 7.8% of the total hours is hardly an issue. I should have done further math and noted that, if students take 60 standardized tests over 12 years - using the most outside estimate - they spend, at 4 hours per test, 240 hours, or only 1% of their time on these tests that educators like Perl and Engel hate so much. If we stipulate a 180-day school year, we see that less than 2% of the time is spent on standardized tests.

One might argue that it's not the amount of time spent on the tests, but the effect that the tests have on curriculum, that is the issue here. But then, why throw in these unsupported statistics?)

I wonder what kind of readers our children will be when they graduate, having taken dozens of tests instead of having read dozens of novels. What kind of writers will our schools generate when students have spent all of their time answering short-answer questions instead of articulating their ideas? I want more from my child's education than "proficient" test scores. Our children require more than a factory approach to schooling.

I agree entirely. But if my child's teacher were to forgo novels and creative writing entirely in order to do nothing but test prep, I'd have more complaints about the teaching style and the curriculum than the tests. Nothing about CSAP requires that teachers stop assigning novels and essay assignments, and I find it hard to believe that students who master tough novels would have a hard time on the exam.

A Colorado school district recently determined that dinosaurs would no longer be part of the first-grade curriculum. What could possibly be wrong with 6-year-olds learning about Tyrannosaurus Rex, you ask? Isn't the excitement of dinosaurs the reason we didn't drop out in the first grade?

I think I was more excited about ghosts, but still, what does this have to do with the tests? And that's one school district out of...how many in the state?

Jefferson County schools and the majority of districts throughout Colorado are implementing a new curriculum modeled after the CSAP - which also is used to rate schools - and pushing out things like dinosaurs.

Did I miss the memo about dinosaurs being essential in order to teach reading skills to first-graders? And while concern about classrooms changing to test prep curriculum is warranted, again - why is the test being held responsible when schools have decided that they cannot teaching reading, writing, and math effectively without help from McGraw-Hill?

Supporters argue that the CSAP tests students on important skills. Fifty short-answer, computer- scored multiple-choice questions cannot accurately reflect a year's worth of learning...

Define "accurately", and then show me an an alternate assessment that is this short (if it's 50 items, where are those 21 days of testing coming from?), has high reliability, and can be given this easily to every student in the state at the same time. There isn't one. Other forms of standardized tests (non-multiple-choice items) would be more expensive, more time-consuming, and less reliable. And non-standardized tests, which include teacher grades, means that student results cannot be compared between schools and between districts. Standardized testing is the least worst method of assessing every kid for comparison purposes.

The first time I wanted to become a scientist was in the sandbox during an palentological dig for dinosaur fossils (which I later discovered were chicken bones). My kindergartner will not find this same joy next year when she enters the first grade. She will miss the lessons on herbivores, carnivores and omnivores. Her instruction will not include ecosystems or the food chain. She will not come running home with questions about volcanoes or giant meteors. Shoeboxes filled with dirt mounds, dead plants and plastic dinosaurs will be replaced with worksheets crammed into notebooks.

Worksheets about what? Did Colorado decide that lessons must be topic-free and that any discussion of science is illegal? The kids will be learning about something; otherwise, how will they learn to read?

I'm not trying to be mean; Ms. Engel is described as someone who has worked a great deal with Denver's inner-city students, and I'm sure she's as concerned as all get-out about their education. And if Colorado has indeed removed all interesting content from lesson plans, she has reason to be concerned. But I doubt that's what Colorado has done; even if they did, would removing the tests mean that the curriculum would automatically revert to something interesting and effective, for every kid, at every school?

Update: Someone in the know in Colorado sent more information on recent Colorado testing results (here and here). My reader also points out that only one school, Cole Middle School in Denver, faces conversion to a charter this year if scores don't improve, and if so, it'll be the first school to have done so. Hardly the "chaos" that Perl envisions.

Speaking of Perl, my reader also pointed out this article, which says that three years ago, Perl was suspended for refusing to give the tests:

A teacher in Greeley was suspended for six days for refusing to give the tests to his students. Donald Perl said the tests violate Colorado's constitution and put non-English-speaking students at a disadvantage because most of the tests are in English.

Fill in the bubbles, win some money

Hmm, when does motivation become bribery?

Frankly, Deering High School junior Samantha Webster felt bribed. Just showing up to take the state's standardized assessment tests was enough to get her into a raffle with a top prize of $100.

Since the federal government last year began to use test results to measure school quality, many Maine schools have decided to go beyond serving orange juice and muffins to test-takers. Now they're copying schools in other states and offering rewards ranging from raffle prizes to pizza parties - even a day off from school.

I don't think this is an indication that schools are "test-obsessed." Motivations like free food and free leisure time work as well on high-schoolers (if not better) than they do on adults, and these are pretty low-key. Sure, some kids will come to win a chance for pizza and still draw a Batman logo on the bubble sheet, but the kids who would have been willing to show up anyway will probably appreciate the possibility of an extra reward:

Even though Deering met "adequate yearly progress" under the federal law last year, Roy said there is always room for improvement. To draw the best performances from the 260 or so juniors taking the test this year, she moved testing out of the school gym and into classrooms. She also authorized two raffles using proceeds from the soda machine in the faculty lounge as prize money.

"We felt it was appropriate to use that money as a marketing ploy to help our students give us their very best effort," Roy said of the drawings, each of which will feature one $100 prize and probably eight $50 prizes. "I guess we feel it's a worthy investment if we truly get results that reflect what the kids can do."

The schools are under pressure to get as many kids as possible to take the exam, and the kids know the exams don't affect their grades. And if individual scores aren't returned to students, then there's not even the incentive of useful feedback. So why not a little external motivation? The idea of paying people to take tests isn't unheard of; post-K12 pilot exams often involve financial reward for those examinees willing to show up and take the pilot tests seriously.

Third-graders running the gauntlet in Wisconsin

Wisconsin's third-graders are getting ready to sharpen, read, and bubble:

Sometime in the next three weeks, nearly every public school third-grader in Wisconsin will be sharpening his or her No. 2 pencils. Elementary schools are giving the Wisconsin Reading and Comprehension Test to determine if students are reading at the third-grade level.

"What happens is, of course, you're compared to everybody else in the world," Zanotelli said. "This one day is a reflection on supposedly everything we do all year long."

Well, yes. I see this sort of complaint often, and the erroneous assumption behind it is that, even in a good school, every single kid could have a bad testing day on the same day. Test scores do contain error, but that error goes in both directions, and chances are that on any one day, some kids will score worse than expected, and some kids better. A school that's doing a good job is, even with testing error, more likely to have a high mean than a school that's doing a poor job.

Like schools, parents pay attention to the tests.

"I think our parents here in Marshfield are very astute in the fact (that) reading is very important to schooling and lifetime employment," said Bernice Lansing, director of curriculum and instruction for Marshfield School District.

Wow. Not much gets by those Marshfield parents, does it?

Often parents will ask teachers and counselors how they can help their child prepare for testing.

Even so, students should get a full night of sleep and then eat a good breakfast in the morning the days of the exam, Stone said. Greenwood, and many other districts, are making sure students have a snack either before or just after the test to keep their energy levels up.

That's pretty good advice, and this scenario - an untimed test that results in public data but no sanctions - is a pretty good scenario for third-graders. Best to catch those problem readers early.

The good, the bad, and the FCAT

The Lee County (FL) League of Women Voters hosted a seminar for discussion about the good and bad aspects of the FCAT:

The FCAT is a needed accountability tool but has become “sensationalized” and “politicized,” Lee County School Board Vice Chair Elinor Scricca told the Lee County League of Women Voters. She joined Richard Itzen, director of the district’s Department of Evaluation, and Florida Gulf Coast University assistant professor Diane Schmidt in discussing the merits and flaws of Florida’s state-mandated standardized test...

“I never gave my students a dollar for an A. They were expected to get that,” she said. “We have put money in the place of motivation.”

I find it hard to believe that her students are working hard to earn more money for the school district, rather than to benefit themselves. On the other hand, if students interpret more money as more classes, more extracurriculars, and more opportunity for them to get a leg up on college, what would be wrong with them being aware of the financial situation?

League leaders invited the educators to their meeting because the group is preparing to take a statewide stance on the FCAT this spring. Itzen acknowledged there are flaws in the system but argued the FCAT has had a positive effect on public schools: Teachers know what skills they should be teaching, and the test-makers know what is on the schools’ curriculum...

But Schmidt worries the pressure put on teachers and students to score well has taken the focus away from the test’s purpose. Instead of encouraging integration of basic skills in all lessons, teachers are replacing creative, thought-provoking lessons with FCAT practice tests, she said.

It sounds to me like those teachers have been brainwashed to believe that mastering basic skills and being creative are mutually exclusive, and it's helpful that the test is revealing that.

Kids are being kept out of art, music and gym classes for remedial reading — and the pressure follows them home, she said.

I'm sorry they have to miss out on art, and I myself would have quit school if I'd have had to quit band in high school. Then again, I already knew how to read. It's not a legitimate criticism of the test to blame schools for focusing on low-performing students by requiring those students to spend more time on basic skills, and less time on extracurriculars.

Itzen recognizes pressure — especially on third-graders. The state’s requirement to hold back those who do not score high enough is difficult, but schools cannot just pass kids through the system, he said.

“Retention is not a fun thing ... (but) being promoted without having the skills to succeed out in the real world is worse,” he said.

Be interesting to see what "official position" the League of Women Voters takes on the FCAT.

Testing teachers, estimating quality

Michael Friedman tackles the topic of whether testing produces better teachers:

It seems obvious - impose qualification tests on teachers and you will improve teacher quality, thereby improving teaching. Unfortunately, however, according to an NBER report last year, it doesn't work.

Does Teacher Testing Raise Teacher Quality? Evidence from State Certification Requirements by Joshua Angrist and Jonathan Guryan. [Note - I don't have a subscription, so I'm going off the summary and Friedman's comments.]

Angrist and Guryan look at teacher quality and compensation in states and districts that require and do not require certification testing.

Obviously (although not examined by this study), certification testing eliminates some of the very worst teachers. Logically, by reducing the supply of possible teachers, certification testing increases salaries for teachers. The effect for teachers with BAs by an average of 3.3%. Disturbingly, certification testing, by increasing the costs of being a teacher, disproportionately discourages the very best teachers who have other options besides teaching.

Angrist and Guryan believe this type of impact is "ambiguous," and I agree. I find it hard to believe that certification tests would be more likely to discourage the best, rather than the worst, teachers from going into the profession, but I suppose it all depends on how one defines "best" and "worst."

In the Angrist and Guryan paper, teacher quality was estimated by average SAT score of the undergraduate institution from which each teacher graduated. I find this a very odd estimate of teacher quality, unless one is willing to assume two things. One, that a teacher's competence in the classroom is very strongly correlated with the average SAT score of their institution, and two, that the SAT scores of students in education programs are comparable to the SAT scores of their instituations as a whole. But students who enter education programs tend to have the lowest SAT scores of any students; only those planning to major in Public Affairs or Home Economics have lower averages. Thus, the second assumption isn't valid. It might still be the case that teacher quality is correlated with institutional SAT mean, but I've never seen any research to support this assumption.

When using this estimate of teacher quality, what Angrist and Guryan apparently found is that state-mandated certification testing didn't necessarily identify the teachers who came from the schools with the highest SAT means. Again, this doesn't really mean anything unless we believe in this definition of teacher quality, and I'm skeptical about that.

Friedman has this to say:

Angrist and Guryan recognize that this result is not very helpful. One possibility is to set a floor on teachers' SAT scores. I can hear Kimberly Swygert reacting already... The SAT score has never been validated to confirm whether teachers with higher SAT scores teach students better.

True, but my reaction to this idea isn't as negative as Friedman assumes. Yes, the SAT has not been validated as a measure of overall teacher quality, but it's certainly a measure of academic aptitude. It would be expected (if as yet unverified) that potential teachers with higher SAT scores would have a stronger grasp of academic concepts and core subject matter. While the grasp of the subject matter is not the only element that goes into creating a competent teacher, it would not be unreasonable to perform a study that calculates a different measure of teacher compentency, and then correlates it with SAT scores. I'd be very surprised if those teachers who had SAT scores below a certain level were found to be competent by any measure, no matter where they got their degrees or how much they loved dealing with kids.

Another possibility is to increase the rewards for the best teachers by offering higher salaries in their first few years to teachers who do particularly well on certification tests. That could reduce the incentive for the best teaching candidates to find other occupations.

True, but I still believe that it's the worst, not the best, teachers who are the most likely to be scared off by the idea of a certfication test. And by "worst" here I mean both those teachers who performed very poorly on the SAT, and those who graduated from ideology-heavy, anti-testing, "progressive" education programs in which fostering self-esteem is more important than fostering academic progress.

Whatever the answers are, however, this study is very disconcerting to people like me who instinctively look to standards based initiatives to improve the quality of schools and students. I have no problem if testing raises teachers' salaries if it also improves the quality of teachers. If it just raises salaries then what is the point?

Again, I didn't reach the same conclusion, because I disagree with this study's estimate of teacher quality. I believe that standards-based certification tests can and should be used, and that teachers should be required to demonstrate skills in the subject area they teach, in addition to their classroom-management and "theory of education" knowledge.

Fighting (and preparing) to be Number One

In New Jersey, the battle rages on between local rivals Montgomery High School and Princeton High School for - a basketball championship? A football trophy? Nope, the battle is for the best combined SAT average in the state:

Montgomery High School has overtaken perennial local powerhouse Princeton High School in SAT performance to become No. 2 in New Jersey. And by this time next year, Montgomery school officials say, they're all but assured of capturing a state title that they covet with the same passion that some schools confer on football and basketball championships.

There's no denying the success of Montgomery's SAT preparation machinery after a 36-point climb for a combined average math-verbal score of 1220 for 2002-03. That's nine points behind two-time state leader Millburn High of Essex County.

What preceded that 36-point climb? Test prep programs in which juniors and seniors practiced one hour a day, five days a week after school for a month, and then increased to three hours of practice a day right before the test administration.

Only 70 high school students got perfect scores on the SAT last year, but 4 of them came from Montgomery High: Kevin Chen, Shinn Chen, Xiaojing Huang, and Amy Chen (none of the Chens are related).

What's more astounding that the fact that the program works is that students seem to be enjoying it:

Mike Laskey, a senior who scored a 1560 and wants to go to Boston College, said the program works because it is fun. "You could pay much more for the Princeton Review or another SAT program, but this one is free and improves your ability more," Laskey said...

"You ever heard the saying, `Sugar helps the medicine go down easier'? Well, this class is just that," Blitzer said. "Pop culture is a tool to learning and identifying some of the hard vocabulary."

And speaking of New Jersey, their DOE released school report cards yesterday. Here's a summary of related articles:

"Standardized test scores encouraging for region" - Philly Inquirer
"Educators upbeat on test scores" - Courier Post Online
"Milville highest on SATs" - NJ.Com
"Report cards: Average marks" - NJ.Com
"Several districts show improved SAT scores" - Courier Post Online

You can view all the report cards for NJ here.

The bigger, the better?

What's correlated with FCAT scores? Library size:

Allie Henderson has a full plate this week. "We'll be taking FCAT math, FCAT science and FCAT writing," said Henderson. Like thousands of other Lee County students, she's tackling the FCAT, Florida's standardized test. The good news - science is on her side.

Fort Myers High School has the only A grade on its state report card based on FCAT and it also has one of the biggest libraries. Only two other high schools in the district got a B grade -Cypress Lake High School and Estero High School. Both schools have at least three full-time librarians on staff and more than 18,000 books...

And it's not just books; libraries often have valuable tech resources for kids who don't have computers at home.

Florida isn't the only state to do this kind of study. At least six other states have also looked at the subject and found a strong link between libraries and test scores.

Not surprising.

A passle of tests in Alabama

No mas, cry survey respondents in Alabama, who think there's too much standardized testing:

Two-thirds of respondents to a new statewide poll favor reducing the number of standardized tests that public school students must take. A slight majority, meanwhile, said they believe that such tests gauge teachers' test-preparation abilities rather than students' actual knowledge, according to the results of the Mobile Register-University of South Alabama poll...

Jo Ann Webb, spokeswoman for the U.S. Department of Education, said the argument faulting educators for "teaching to the test" is invalid. "If you teach a child, then test a child on that information, then I don't understand the criticism," Webb said. "You're still teaching the child."

The majority of those polled still support the high school graduation exam, though, because they believe that really does hold schools accountable for what students learn. And I can understand their frustration with the long list of tests facing Alabama's students:

Depending on their grade level, elementary and middle school students must take some or all of the following tests: Stanford Achievement Test, Alabama Direct Assessment of Writing, National Assessment of Educational Progress (NAEP) and the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) test.

To graduate, high school students must pass all five portions of the graduation exam: math, language, reading, science and social studies. Special education students take the Alabama Alternate Assessment to get a diploma. College-bound students also take the ACT anor SAT admissions exams.

One Mobile County spokeswoman insists the tests are necessary:

"How else can we find out what our children know?" said Nancy Pierce, spokeswoman for the Mobile County Public School System. Pierce said testing allows teachers and schools to evaluate their progress in preparing students for the next grade level.

"If you don't know what subjects and verbs are, how can you go into more elaborate sentence construction? If you can't do addition, how can you go into multiplication and division?" Pierce said. "If students don't have certain concepts, they can't go on."

To me, this statement implies that the sort of objective assessment needed to decide whether a students is ready to progress to the next level isn't being done by teachers in the classroom. Is Pierce wrong in assuming that, or are the teachers' classroom assessments so subjective and/or inflated that Alabama needs a slew of standardized tests in order to tell what's going on?

The cause of Marin's high test scores

The Marin Independent Journal celebrates the test scores of local children, but misstates the relationship between test scores and SES:

Marin students performed better in standardized tests than students in other Bay Area counties and were more likely to take Scholastic Aptitude Tests than their peers, according to a report to be released today...

In the past, national studies have shown that high test scores are, more than anything else, the product of socio-economic status of families, with students from the high end of a continuum consistently scoring higher on standardized academic achievement test than those from the low end. This is borne out in Marin, where a high percentage of parents of students tested are educationally and economically on the high end of the scale, compared to the rest of the state, as well as the nation.

Correlation does not imply causation. Simply because, overall, students who come from wealthier families may do better on tests, that doesn't mean that, as stated above test scores are caused by the wealth of the parents.

It is logical to draw that conclusion in our society, because kids from wealthier families have more opportunities and tend to go to better schools. But their parents might be wealthy because IQ and income tend to be correlated in our society as well, and it might be that intelligence - and learned traits like self-discipline and competitiveness - account for both parental income and child test scores. How many parents in Marin County had to work long hours at hard jobs to earn the money to live there? Who's to say their kids didn't inherit or learn those traits from them?

This might seem like a nitpicky point, but it's not. One of the most important concepts taught in Stats 101, is that correlation between A and B does not imply that A causes B or B causes A, because a third factor C might cause both to occur. This concept is essential for research because missing that third factor can lead to totally erroneous conclusions like "Poor kids have no chance of making good test scores" or "Tests don't measure anything other than parental income," both of which assume that money in and of itself is the most important factor affecting test scores.

At least the article avoids the truly boneheaded statements, and it doesn't go into the nonsense about how it's not "fair" for some kids to have more money than others, but the author still gets it wrong by assuming that correlation implies causation. Congrats to Marin's students, though (who I bet would be rather offended if you told them they only scored well on tests because of Mommy and Daddy's money).

Update: As Devoted Reader Jeff W. points out, I missed the real story:

Marin is the wealthiest county in California. Kids there enjoy advantages that most kids in this country can only dream of.

Yet only 45% were proficient in English and algebra. That's the real story.

According to Wikipedia:

The median income for a household in the county is $71,306, and the median income for a family is $88,934. Males have a median income of $61,282 versus $45,448 for females. The per capita income for the county is $44,962. 6.60% of the population and 3.70% of families are below the poverty line.

Can't blame poverty for poor school performance, can they? Here is the county's STAR score report for 2002. According to this, the scores for Language Arts breaks down as follows:

Grade ----- % scoring Proficient or Advanced
2 ---------- 56
3 ---------- 61
4 ---------- 67
5 ---------- 61
6 ---------- 60
7 ---------- 66
8 ---------- 63
9 ---------- 58
10 --------- 55
11 --------- 52

For each grade, the percentage scoring only Proficient (not advanced) is anywhere from 24% to 43%. So I'm a little confused. Is the 45% for 2003 reported in the article above for only the Proficient category (which would reflect a small increase) or for both Proficient and Advanced together (which would reflect a large decrease)?

Either way, Jeff is right. I missed the real story, which is (a) that in 2002, with the data I'm reporting here, only 52% of 11th-graders in a very wealthy county scored at or above Proficient in Reading, and (b) sad as that is, they're still doing better than the country at large, because 57% of Marin's 11th-graders scored above the national 50th percentile mark.

Calling a foul on SAT rankings

Based on the raw numbers, the state of Georgia ranks 50th - last - when it comes to SAT scores. But ranking states by SAT scores is not a wise thing to do:

Mark Musick, the president of the Southern Regional Education Association, agreed. “Georgia has a problem, it’s just not that we’re 50th.”

In North Dakota and about half the states, the ACT is the dominant test for college-bound students. Colleges can convert ACT scores to SAT scores, and once converted, according to a Southern Regional Education Association list, Georgia ranks higher than 11 other states.

Psychometricians know this, but the general population doesn't hear it enough in the media. In many southern states, all 10th- and 11th-graders are encouraged to take the SAT, whether or not they plan to go to college. This practice helps fulfill the original purpose of the SAT - to identify those kids who might not otherwise know they're college-material - but it's guaranteed to drag the averages down.

The important question is not, "What is a state's mean SAT score?" The important questions are who takes the SAT in that state, how many take the SAT, and what the plans are of those who take the SAT. That doesn't guarantee an apples-to-apples comparison, but it does prevent the comparison of Georgia's general population with the kids from North Dakota who plan to go to Harvard.

(Thanks to Devoted Reader Jim P.)

When tests reveal bad news

The Heartland Institute wonders why exit exams are so unpopular in some circles, when the test scores seem to be of direct use to students:

The advent of high-stakes testing is revealing more than just information on what American high school students know and are capable of doing; it is also revealing a significant shortfall between that assessment of actual skills and what schools have been telling students about their achievement and ability.

For some students, the failure to pass a high school exit exam is the first warning signal they may be sorely unprepared for the demands of college.

A group named the Mass Insight Education and Research Institute, along with the UMass Donahue Institute, completed a study last October entitled, "Seizing the Day." It's a longitudinal study of kids on the "front line" of education reform, and some of the study's conclusions are very interesting; namely, that the high-stakes exit exam (in this case, the MCAS) is what forced many students to work harder and improve their academic performance:

The researchers found most of the 32 percent of students in Boston, Springfield, and Worcester who failed the state’s Massachusetts Comprehensive Assessment System (MCAS) exam on their first try--in 10th grade--were more than a little surprised: Almost 90 percent of them had a C average or better at the time.

By their senior year, 94 percent of these students from the class of 2003--the first required to pass the MCAS to graduate--received a diploma. The study found 82 percent of students who need extra help are now taking steps to get it; juniors are committing earlier to passing MCAS; and at-risk students are putting more effort into their school work because of MCAS.

Sounds great. Who could oppose this?

But the increased effort and success may occur almost in spite of the students’ teachers, who, in addition to inflating grades, are sending mixed signals to the students about the test. Seventy-one percent of the students surveyed perceived negative attitudes towards the MCAS on the part of their teachers...

It makes sense that any teacher who inflates grades will mistrust the test, either because they genuinely believe every kid deserves an "A," or because they believe their classroom grades capture something ineffable that the basic-skills exam misses. Other teachers might oppose the tests because they were taught standardized tests were uniformly bad, or biased; some teachers may resent the time taken away from classwork for testing.

But when grades alone are used to determine which students are ready for college, the results are decidedly mixed:

For example, in Georgia, 40 percent of high school graduates who receive that state’s Hope Scholarship are losing it after about a year because they can’t keep up their good GPAs in college.

A recent report from the National Center for Education Statistics found the number of students taking at least one remedial course upon reaching college has risen to 35 percent from 28 percent five years ago. At the same time, the percentage of college-bound students carrying an A average has grown from 28 percent 15 years ago to 42 percent now, according to the College Board.

The Nevada report is particularly blunt (not surprising considering Fred Phelps, an acquaintance of mine, is on the author list):

The problem of poorly prepared students entering college is growing, as far more students are now going to college. In recent years, approximately half of Nevada's high school graduates entered college. Last year nearly 10,000 of them were enrolled in remedial courses. In too many cases, they needed help with knowledge and skills that should have been learned in the third or fourth grade...

At the heart of Nevada's problem is the issue of teaching philosophy. Generally, teachers - especially elementary teachers - are taught to think of teaching and learning as a process that follows student interests and inclinations - whether or not it leads to the achievement of curricular objectives. These teachers are trained to design learning experiences that optimize student interest and enthusiasm, not particular learning results. The consequence is that many students simply acquire a patchwork of knowledge and skills - often with significant gaps and weaknesses. Similarly, many never learn that dabbling in schoolwork is not enough - that success requires meeting challenges and overcoming them.

Oh, and that cult of self-esteem so valued by some teachers? Nevada's college professors were fairly dismissive of the value of training in "cooperative study activities, self-esteem enhancement and work with diverse students" in a college classroom. What was considered to be important? "Ability to focus in class," "Effective study habits," "Ability to complete work on time," "Patience and persistence in completing course," and "Understanding "work ethic" related to education." The professors tended to be in favor of stronger standards (although unconvinced that testing could help reveal the effects of them).

The report concludes:

...our professors found their students to be weak in both the basics and higher-order skills, and currently they see no trend toward improvement. Thus, it seems fair to conclude that the critics' apprehensions are, at the least, misplaced. Apparently there is little danger that the imposition of standards and accountability will make matters worse.

In other words, things are so bad for college freshmen that it's hard to argue that adding new standards and accountability measures will make them even more ill-prepared for college.

Putting the little ones to the test

Lance Izumi of the Pacific Research Institute for Public Polic has an article on the OC Register online that asks the question, which do we value more in elementary students - self-esteem, or learning?

Remember when Pinocchio was led astray by the fast-talking fox who promised him fun instead of the hard work of school? By the end of the story he turned into a donkey. Some modern-day foxes in Sacramento want to raise student self-esteem by eliminating state testing of second-graders, which could result in an education for our children better suited to donkeys.

Legislation authored by Assemblywoman Loni Hancock, D-Berkeley, and sponsored by the powerful California Teachers Association, however, would exclude second-graders from the testing requirement.

Why? Because of the belief that testing is too stressful for second graders. I must note that this attitude is not inconsistent with the "conventional wisdom" I learned in graduate school. We assumed that testing below the third grade would not be useful because the test takers wouldn't understand the stakes and wouldn't necessarily be able to focus on the test in order to give useful results.

Our concern wasn't so much that we thought second-graders would irreparably harmed by a high-stakes test, but that they would goof around and be unfocused and not really get the point of it all.

Mr. Izumi doesn't agree with the stress argument:

Poor performance by a child on the state test isn't the end of the educational process, but the beginning. Test results allow teachers, administrators and parents to see student weaknesses and target remedial help.

...if testing were conducted beginning in the third grade with results not coming in until the start of grade four, a student would be halfway through elementary school before this valuable information became available.

Assemblywoman Loni Hancock's suggestion of using textbook end-of-chapter tests as in-class assessment fails the standardized requirement. Students could not easily be compared to one another, or tracked over time. I tend to agree with Mr. Izumi that early testing might reveal early problems in enough time to get students back on the right track, but thanks to what I learned in graduate school, I believe that it's not necessarily a simple matter to get good, solid, reliable data from kids that young.

There's definitely a dearth of research on testing children below the third grade, and it might be the case that the type of test would have to be qualitatively different in order to validly and reliably assess such skills as reading in children who are just beginning to learn what testing is all about.

Cornell defends the SAT

Another rarer-than-a-dog-bites-man story: Defending the SAT, in the Cornell Daily Sun.

The SAT periodically appears on the national radar. Recently, the film The Perfect Scorereminded us to what lengths high school students will go to get that elusive 1600...

Embellishments aside, the film shows just how skewed and misguided the prevailing wisdom on standardized testing is. Brian Robbins, the director of the movie, was quoted in a recent Sun article as saying: "I've often questioned the ethics of standardized testing, and I feel that the importance placed on SAT scores has too much of an effect on a kid's future." What special insight Robbins -- who specializes in teen sports movies -- has into the SAT is unclear.

Ah, I'm lovin' this article already. Not only for its pro-SAT stance, but also because it's willing to poke at people who pretend to be experts in subjects about which they know very little.

Were college admissions offices to actually stop requiring SAT scores...increased emphasis would fall on more biased measures instead. The other main components of a college application -- GPA, admissions essays and recommendations -- are all highly subjective methods of evaluation...Relying more on these tests than on the original SAT puts students in lower-performing schools at an even higher disadvantage.

...sadly, the public debate over the SAT is largely based on fantasy and fabrication as well. As painful as it may be, the SAT is the only way to keep college admissions fair.

It may seem like a cliche when SAT-supporters turn the focus back around to the quality of public education, but that's where the true crux of the matter lies. The SAT is but a measuring stick, and removing the SAT does nothing to make students more prepared for college, even if the lack of a standardized test is more likely to result in the admission of some students to college. Testing critics almost always confuse the message with the messenger.

(Reform K12 has more.)

How volunteers help raise test scores

In Baltimore, community volunteers help students with low test scores read more quickly and memorize their multiplication tables:

The program involves community volunteers meeting with pupils three to five times a week for about 15 minutes to practice reading "sight words" - words that pupils should be able to read without sounding them out. They also help pupils improve reading fluency - the accuracy and speed at which a child reads - and memorize multiplication tables.

The second technique, Repeated Reading, entails having children read passages from books that are familiar and unfamiliar. Volunteers count the number of words read correctly and use a graph to chart progress.

As for the math? Oh, they used flash cards. Some "progressive" types might hate "drill-and-kill", but a kid who hasn't developed an automatic understanding of the basic math skills simply won't be able to progress any further.

Controversy over New York's third-graders

New York's grand plan to end social promotion of third-graders isn't going over too well with some parents:

Parents' groups are speaking out against a plan barring third-graders in public schools from advancing to the next grade unless they pass standardized tests.

"Our position is 'no' on retention. It's punitive and unfair," Robin Brown, president of the group, told the New York Post for Wednesday editions. She said parents were angry that Bloomberg had adopted the plan without first soliciting their comments...

An NYTimes Op/Ed has this to say:

...the administration risks frittering away its political capital by reinventing the wheel and rediscovering educational policies that failed miserably in the past. This seemed clearly the case in the mayor's State of the City address, which unveiled strict new promotion standards that would cause as many as 15,000 children each year to repeat third grade — or roughly four times the number that are held back today.

Mr. Bloomberg has told New Yorkers time and again that he wants them to judge him on how well he does with the schools. He stands a much better chance of doing well if he listens more closely to his critics and takes a less combative attitude toward the union.

To be, or not to be? That is the (standardized test) question

The SAT writing component comes in for a bit of bashing, good-natured and otherwise, from testing critics:

Shakespeare's heralded writings wouldn't be sufficient to get him into Dartmouth or any other Ivy League institution -- at least not according to the Princeton Review. The Unabomber, Ted Kaczynski, on the other hand, would have his pick of the Ancient Eight.

"What fools these College Board people be," said John Katzman, co-author of the article, and CEO of the Princeton Review. "They simply tacked the essay from the old SAT II Writing test onto the new SAT in order to appease its largest client, the state university system of California, which was threatening to stop requiring the outmoded SAT and find a more relevant test."

So let schools stop using an assembly-line procedure to admit students, and applicants can write more sophisticated essays specifically for each university. The point here seems to be that the College Board is deliberately producing a superficial assessment of writing, but with two millions examinees a year, it's hard to see how they could administer and score anything else.

The claim that the SAT would give poor grades to Shakespeare and Hemingway is amusing, yet irrelevant. The SAT writing assessment measures mastery of basic skills; while a student who writes like Hemingway might perhaps fail it, I'm willing to bet that most of the failures are students who don't have Papa's grasp of the English language. If the new SAT Writing section does a good job of snagging those students who slept through four years of English in high schools, colleges will find it useful. If the assessment doesn't work, schools will stop using it.

And the part about the Unabomber is pure sensationalism. Theodore Kaczynski was admitted to Harvard at the age of 16 and did his graduate work at the University of Michigan. He may be crazy as a bedbug, but he wasn't dumb, so it doesn't surprise me that his writing was lucid (even his most threatening letters are now part of a scholarly collection at the University of Michigan).

Are California's juniors ready for college?

Rarer than a dog-bites-man story: A call for more testing of high school students in The Desert Sun (Palm Springs, CA).

Coachella Valley educators have the right idea but don’t go far enough in planning to give high school juniors a mini-college competency exam. The intent of the test is to raise red flags, to help students determine whether they need additional preparation in meeting the college-level English and math requirements.

An annual student proficiency report for first-year freshmen entering California State University in fall 2003 showed that about 58 percent needed to take remedial courses to meet college standards in math or English or both. Algebra and reading comprehension are among the key skills on which students need to work.

The tests are apparently called the Early Assessment Program, and they're designed to assess whether students are up to college-level English and math requirements. This document has more information about the test that is a joint effort of public schools, the State Board of Education and Department of Education, and California State University. The webpage about the project is here. This article says the test will be available for juniors in the spring of 2004.

No information online about item types yet, though. If anyone has any further information, let me know.

Teaching to the test in Philly

Philadelphia's response to lower test scores? More Kaplan-designed test prep:

The school district told its 11th-grade English and math teachers that they must spend 20 minutes a day for the next 10 weeks using a test-preparation program developed by the New York City-based Kaplan - an educational-services company perhaps best known for its test-preparation programs.

The district's new program is designed to prepare students for the Pennsylvania System of School Assessment, to be given in late March. Test prep is common in suburban districts, and school officials more frequently are using software programs to help students prepare.

Needless to say, some are unhappy with this idea:

But some teachers are questioning the way the program is being rolled out; they worry that it is sapping the creativity from their teaching and interrupting their traditional course work. They say it is interfering with term papers and the study of literary works. One English teacher described it as "practically reading from a script."

Some schools use block scheduling with 90-minute periods; the test prep accounts for less than a quarter of their class. But other schools have 45-minute periods, which means nearly half of the class is spent on the Kaplan program.

In response, one principal says that in these days of high-stakes exams, teachers must be comfortable with "teaching to the test." Not only is that a not-very-sympathetic response, it also shouldn't always be the case. "Nearly half the class" is a big chunk of time to spend on test prep.

I like the idea of having practice tests available, and I think teachers should use that feedback to tinker with teaching plans (although, why aren't they doing something similar already?). I can understand why teachers don't want to surrender their lesson plans to Kaplan, though.

Reform K12, not surprisingly, has more:

One of the direct side-effects of high stakes testing is that the act of testing is becoming so important, it's starting to cause folks to forget that that teaching and learning are what schools are for, not test-prepping.

In our view, standardized tests are intended to measure learning, not test-prepping. Thus the Philly program, while it may well boost scores, will simply serve to mask the real problem.

Duck season! Wabbit season! FCAT season!

The standardized testing times in Florida have begun anew:

Ready or not, it's FCAT time again. The Florida Comprehensive Assessment Test begins Tuesday with the writing portion, which fourth-, eighth- and 10th-graders take each year. FCAT reading and math tests are less than a month away.

I'm wondering if this principal is working miracles, or if she's going to end up with a whole bunch of kids who write like Irvine Welsh:

Allison Kalbfleish, principal at Stambaugh Middle in Auburndale, said her whole school works on writing. She said the children learn to form a traditional five-paragraph essay with an introduction, body and conclusion. She said they are encouraged to use colorful language and vary their sentence patterns and word choices.

She doesn't think that the rigidity of the five-paragraph essay stifles creativity.

Well, no, it doesn't, but let's hope she and her middle-schoolers both have the same definitions of "colorful" language.

After writing comes the math and reading tests taken by students in grades three through 10. Those scores, along with the FCAT writing scores, are the primary factors that determine what grade a school receives from the state.

"They can actually write on the test," she said. "That's one of the big challenges. They're not used to writing on their materials in class or inside their books."

Good point.

Elementary schools make sure children know how to properly bubble in an answer sheet. But they also have to deal with the anxiety and even fear that seizes some children. Teachers say they really try to encourage their students.

"I always tell them it's not worth tears and tummy aches," said Kittleson, the Cleveland Court teacher.

Also a good point. Encouragement, along with the most effective instruction, is probably the only way to reduce test anxiety.

Of course, when kids don't pass, the parents become anxious too:

Jay Bowen has a son who did everything he needed to do to graduate from high school last year, except pass the 10th-grade FCAT. He dislikes the highstakes nature of the FCAT.

"Why should my son have to go get a GED when he met all of the requirements to get a diploma?" he asked.

Emphasis mine. He's right, but not in the way he thinks he is. Why should his son have to go get a GED? In 12 years, his son couldn't manage to master 10th-grade material. His son might have met all the other requirements, but he performing at a level well below what the school says he should be at. And Mr. Bowen should be asking questions about that.

It's not that I don't sympathize with Mr. Bowen, but when his son completes twelve years of a schooling and then fails the reading portion of a 10th-grade exam six times, he should asking for more than copies of the test to help him pinpoint where his son's weaknesses are.

When smartass students and standardized tests collide

What happens when a high-performing senior makes a point of not taking a standardized test very seriously?

Jake Bogdanovich's performance on a recent standardized test has earned the high school honor student some recognition he didn't bargain for. Bogdanovich, who has a 3.8 grade point average, admits he had no interest in the test designed to measure the success of an academic program at Garfield High School...

So Bogdanovich did something out of character for an "A" student: On the bubble answer sheet, he made a tic-tac-toe pattern; he drew characters from the television show South Park in the short answer section; and he took part in a coughing fit that became contagious for other students.

Now he is paying the price. Beginning on Monday, he will serve a three-day, in-school suspension. His status with the National Honor Society is in jeopardy and he lost his position working in the school office.

Emphasis mine. I think Jake's in trouble not just because of his artistic attempts on the test, but because he disrupted the testing environment. It's one thing to just turf on your own test; it's another thing to bother students who perhaps didn't appreciate the little "contagious" coughing fit.

The test at Garfield, as well as Akron's other high schools, is designed to study whether seniors at each school are performing better than the class before them. Bogdanovich was one of 100 Garfield seniors randomly picked to take the test...

After taking three minutes to fill out the last section -- which was supposed to take 25 minutes -- a teacher reported both students to Decapua's office after both said they did not try.

I think Jake's also being punished for his attitude; he apparently thinks it's smart to state that he only makes an effort when there's something in it for him. I don't think the school is going overboard with the suspension, although officials should make the punishment for goofing off clear beforehand.

To whom do good test scores belong?

The Wall Street Journal reports on an odd phenomenon, although it's perhaps not so odd, given the heavy focus on test scores. In Ohio, the scores of gifted children are being credited to their neighborhood schools - regardless of whether the little geniuses actually attend those schools (thanks to Devoted Reader Michael for the link):

Matthew Benton, a self-possessed sixth-grader with an "A" average and an I.Q. of 132, is likely to pass the Ohio Proficiency Tests next month with ease. But his prowess on the tests, which are used to assess schools' performance, won't help Bennett Elementary, where Matthew is in a citywide program for academically gifted students.

Hoping to boost their overall results, schools are squabbling over who gets to claim the test scores of gifted students. In Ohio, the scores of gifted children are credited to their neighborhood schools -- even if they actually attend other schools...

This plan is being considered in other states, by the way. In at least one California district, schools without gifted-student programs complained, and the scores from all schools are now reported as belonging to the district as a whole. Why would this odd situation have come about?

Educators contend that some of America's brightest young minds, including minority and low-income students, are being discouraged from gifted-education programs...

The importance of the scores of gifted children to the survival of schools and the jobs of principals and teachers can lead to statistical finagling that distorts school performance. Ohio's policy, for instance, inflates rankings of neighborhood schools at the expense of schools that house district-wide gifted-education programs.

Ironic, considering that test scores are much of what help classify children as gifted in the first place. Schools want kids to do well on tests, but when students who do really well are identified, they could hurt the school by leaving for another that has a program better tailored to them. So the increased emphasis on test scores for schools as a whole have made schools more protective of their gifted youngsters.

Marjorie Fox, president of an independent foundation that helps the San Diego district attract bright second-graders from low-income families to gifted-education classrooms at different schools, says half a dozen principals have asked its recruiters to stay away from their students...

Principals argue that even gifted kids would do better to stay in their neighborhood schools, but parents counter that kids deserve the chance to move to a school where they are challenged. So Ohio's compromise was to assign test scores to neighborhood schools, even after the gifted child has moved to another target school with special programs. The low scores of special education students are distributed in the same fashion.

Like many compromises, this one seems to satisfy no one. Schools that house many gifted kids appear to be performing poorly, and schools that have never educated the gifted appear to get credit for doing so. And while kids will benefit by being more likely to be recommended for gifted programs, they are understandably bewildered as to why a school they don't attend gets the "credit" for their performance.

New tests for Montana

Montana students will soon be taking a standardized test that is half performance assessment:

In the usual fill-in-the-bubble standardized tests, students don't have to draw a graph tracking global warming, or write an essay about how a hurricane is formed. When a new kind of standardized test hits the desks of Havre fourth-, eighth- and 10th-graders this spring, nearly half the possible points will come from questions like those, to be answered in the students' own handwriting.

Will students be exposed to material suggesting that the global warming controversy is overhyped? Can they write essays that cite Bjorn Lomborg and still get high scores? And wouldn't learning about tornados be more useful for Montana residents than hurricanes? Could an acceptable essay about hurricanes begin with, "Well, we don't get many hurricanes in Montana, but..."

Okay, I'll be serious now.

The new tests require a specific level of mastery in each subject area. The standardized tests students take now, the Iowa Test of Basic Skills and the Iowa Test of Education Development, score students by comparing their scores with those of students across the country in a given year.

That is, they're moving from a norm-referenced test to a criterion-referenced test. And with the addition of open-ended questions, there's certainly more face validity, but the issues of reliability and scoring get trickier (and more expensive).

In constructed-response reading questions, students read short passages and write short essays on them. A sample fourth-grade question asks students to read a two-page passage about a debate over school dress codes and uniforms. They are then asked to "describe how school clothes can create problems for students. Use information from the article to support your answer."

In a sample math test, fourth-graders were asked to make two different patterns of numbers based on rules they make up, and to tell how many 1-inch cubes would fit inside a box with given dimensions.

The reading section sounds fine, but I'm leery of mathematics assessments that involved a lot of writing, simply because writing ability becomes confounded with math ability on these types of tests. When the math assessment is primarily "story questions" and the results depend on writing and labeling as much as on addition and subtraction, kids who are good at math, but not at English, are going to be at a disadvantage.

NAEP scores and Simpson's Paradox

Jay Mathews is back in the WaPo with an article on what the media are missing when reporting test scores, and a good explanation of Simpson's paradox:

Mention Gerald W. Bracey's name in any assemblage of educational pundits and you will often hear an awkward silence...Bracey has often offended self-appointed experts like me by exposing us to the truth, and he is rarely invited to any of our parties.

As Bracey explains, we are overlooking two important factors: (1) minorities make up a much larger portion of the total testing population than they did before, and (2) although they have shown significant improvement, their averages are still relatively low. When you add more low scorers, even if they improve over time, you are not going to see much improvement in the overall average.

In other words, when you're lumping together all test-takers, and you see only a tiny rise in the overall average, that doesn't mean that, within every subgroup, there's only a tiny rise. If there are big differences in both sample sizes and peformance in subgroups, as there is on most standardized tests, then the disaggregated data will tell a much different story - sometimes, the opposite story - than the dataset as a whole (which is why, in statistical analysis, you always disaggregate your data).

Howard Wainer, who is a very well-known, well-respected, and prolific psychometrican, has published a great deal on several statistics paradoxes, including the Simpson's one. Here's an article from 1994 in which he examines NAEP results for black and white test takers:

[For the 1992 NAEP 8th grade math assessment] Nebraska's average score was 277 New Jersey's average score was 271. On the face of it, it appears that 8th grade students in Nebraska do better in mathematics than their counterparts in New Jersey. We note further however that when we examine [mean]performance by ethnic group we find:

If the question is, "I want to place my child in school in either New Jersey or Nebraska. In which state is my child likely to learn more mathematics?" The standardized scores give the right answer...If your child is White, he/she is likely to do better in New Jersey. If he/she is Black, he/she is likely to do better in New Jersey...Presenting the data in a disaggregated way allows these sorts of questions to be answered specifically, but if a single, overall number is needed to summarize the performance of a state's children, for questions like this, one must standardize.

Anyway, back to Jay's takeaway argument:

You can argue that the failure of the white students to improve significantly is a matter of concern, but it is also clear that we have been obscuring the good news about minority score improvements by focusing so much on lack of change in the aggregate scores.

Testing changes in Arizona

Arizona will be combining two standardized tests to give students more classroom time - but might also be lowering the standards on the eighth-grade math exam:

The Stanford 9, given every year to measure Arizona students against their counterparts across the country, and the Arizona Instrument to Measure Standards will take just one week rather than the two it takes now. The tests are given in the spring.

Eliminating classroom time to prepare and take the Stanford 9 will allow students to do better without school officials' losing the ability to gauge their knowledge, supporters of the move said...

So what about that new standards on the math AIMS?

Also Monday, a new scoring system for the eighth-grade AIMS math test was proposed because state Education Department officials believe the test scores do not accurately reflect student achievement.

I find this statement confusing. A change in the standard - in this case, by dropping the required passing scaled score from a 78 to a 72 - means that more students will pass. I'd like to know why Arizona believes that more students passing is a more accurate reflection of achievement. If the standard is eventually changed, I hope it will be because the Education Department really believes that those student who score higher than 72 are in fact doing "good enough" on math. I hope the change isn't being made simply because the state simply has a feeling that more than 22% of students should pass.

Apollo Principal Conger supports the proposed change in the eighth-grade test. "They took the data and I think they want to make it more successful for more students," she said. "It's not lowering the standard or watering it down - it's probably being more realistic.

"We don't want to disenfranchise kids," she said.

Sigh. Look, if you lower the cutpoint, you lower the standard, period (assuming the item types and difficulties remain the same). Lowering the standard is not necessarily a bad thing - IF there's real data to suggest that the current standard is too high and disqualifies too many students who really do have the skills. Let's not start off this discussion, though, by denying that lowering the standard is in fact what Arizona wants to do.

As for the "disenfranchise" statement, well, my first reaction is, "If you don't want to disenfranchise kids, teach them math." Again, if the previous standard was really too high, in that students who made scores between 72 and 78 really did have the skills to go forward in math, then fine. Lower the standard. But don't talk about "disenfranchising" kids with test scores. It's not a low test score that is the stumbling block; it's the lack of skills being conveyed by that score. If kids who don't understand math well will nonethless be passed under this new, lower standard, the state isn't doing them any favors.

Should college students take an "FCAT"?

More discussion on a graduation exam for college students in Florida:

The board is working on the testing provision, which will be voted on in March. But it approved six measures to evaluate the universities, including the number of minority students enrolled at each school and the amount of research dollars attracted.

The members said they want to make sure students are learning, not just graduating. "This issue is not going to go away," said board member Steve Uhlfelder. "It is difficult. But I believe learning assessments need to stay in the discussion."

I foresee just as much controversy over this as for high school exams, if not more. The exam itself does not sound rigorous:

...a test in writing and critical thinking would be given at the end of a student's second year and administered again at the end of the fourth year. In some cases, separate tests in a student's particular field of study, such as chemistry or history, would be given when a student enrolls and again in the fourth year.

However, the same arguments will appear about "teaching to the tests," different "styles of learning," and "multiple intelligences," with the added complication that now the testmakers are going up against college professors, who are known to be a bit more stubborn than their K-12 counterparts.

Oh, look. Already, there's an op-ed published about this, entitled, "Enough Already." You get three guesses as to how the writer feels about this proposed college test, and the first two don't count.

Suing over the LSAT

Here's a short little blurb in the Denver Post about a student who is suing the Law School Admission Council, because she has not been granted extra time on the LSAT. The story contains very little detail, but I can provide some extra information. Not about this particular case, but I can give a good educated guess about what's going on.

The plaintiff, Abby Rothberg from Littleton, claims to have a learning disability, and wants extra time on the LSAT. LSAC has refused to grant her extra time, and thus she claims her presumably-low LSAT scores do not "accurately reflect her overall intellectual aptitude and abilities."

Here's the LSAC page on accommodated testing. There's quite a few links to read on applicants who have a cognitive disability. If you have Adobe Acrobat, you can read the Guidelines for Documentation of Cognitive Disabilities. It's very detailed, and readers are immediately made aware that an accommodation for a cognitive disability is not an easy or simple thing to get. Which is as it should be.

Those who wish to gain extra time on the LSAT due to a cognitive disability must:

* Be assessed by a qualified and licensed evaluator with comprehensive training and direct experience in working with adult populations, who is willing to provide academic credentials and qualifications.

* Show that the current nature of the disability is such that it will have an impact on timed testing, in the sense that someone has the capability to understand the material but cannot respond to it within the proscribed time limit (requests for extra time are the most common). Within the previous three years, tests must have been conducted to show the impact of the disability. Someone who is diagnosed with a learning disability at age 10 and then is never re-assessed will not be able to use that diagnosis alone to apply for an LSAT accommodation at age 21. Anyone older than 21 must have been tested and diagnosed within the previous five years.

* Provide results of a "neuropsychological and/or psychoeducational evaluation," with the expectation that "the assessment will be a comprehensive battery of tests administered by someone with clear credentials in the field (such as board certification by a recognized board)." In other words, a note from your family doctor won't do. Whatever testing is done must be comprehensive and provide a clear diagnosis. A psychoeducational evaluation is also required "without exception" and must be submitted on the letterhead of a qualified professional.

And dig the amount of detail each candidate must provide:

Domains included in each evaluation MUST include the following:

All reports must also include, at a minimum, scores on previous standardized
admission tests, such as, but not limited to, the SAT, ACT, GRE, MCAT, and LSAT, with scores broken down by areas (such as verbal, mathematics, reasoning, critical reading, etc.) and with both the standard scores and percentiles reported. In addition, if accommodations have been granted for any of these tests, the exact accommodations granted and used must also be
described. For example, if you were granted “unlimited time,” provide a report of the specific time used. This information will speed up the processing of your request considerably and will avoid delays due to requests for more information.

I would not be surprised to discover that this is the most stringent set of requirements to be satisfied in order to take a standardized, high-stakes admissions test. Again, there's a good reason for all of this. The LSAT is a difficult, timed test which requires test takers to read lengthy and dense reading passages, answer logical questions, and complete analytical problems (the "logic games"). Being able to do all of this under the set time limit is not unrelated to work in law school. The pool of test takers is competitive, ambitious, intelligent - and litiguous. LSAC has these crystal-clear guidelines in place under the assumption that anyone who does have a cognitive disability will be willing to provide this information, and anyone who is trying to cheat the system won't be able to produce this much proof.

So, back to our plaintiff. I'm guessing she either (a) did not provide sufficient information for LSAC to agree that she had a cognitive disability that warranted extra time, or (b) she had the information but didn't provide it in time to be processed. It's not unrealistic to expect wanna-be lawyers to understand the importance of providing sufficient documentation in a timely fashion, but it's also not surprising to see that a law school hopeful is willing to file a lawsuit. It will be interesting to see how this plays out.

The important facts for new Americans

Here are thoughts on a new citizenship test, from the Cleveland Plain-Dealer:

What does every new American need to know? A government agency is trying to decide. The U.S. Citizenship and Immigration Services held a conference in Washington last week to discuss changing the test that prospective citizens must pass to become naturalized Americans. They hope to have a new version by next year.

Oh boy, now the standardized testing debate gets combined with the immigration/assimilation debate. That's going to get interesting.

The current test, which dates from the 1980s, aims at demonstrating a basic understanding of English and of "the fundamentals of the history, and of the principles and form of government, of the United States." It's not a standardized test but usually consists of about 10 civics questions picked from a study guide of 100. Some of the questions include listing the colors of the flag, identifying the original 13 states and naming the writer of "The Star-Spangled Banner."

Critics say the test focuses on unrelated random facts instead of principles and the meaning of citizenship. They say that it is more important to know the significance of the words "give me liberty or give me death" than to name the person who said it.

Sounds good, but who gets to decide what the true "significance" of those words is? What interpretation do we use? What IS the meaning of citizenship? The reason that "unrelated random facts" get used on these types of test is because those facts are easily learned and fairly objective. I agree that testing the understanding of our nation's history is important, but it's also a much trickier thing to measure than whether or not someone knows when the Civil War began.

President Bush, who announced plans to revamp the test earlier this month, said the United States should set "high expectations for what new citizens should know. Every citizen of America has an obligation to learn the values that make us one nation."

No one can argue with that, although we might hope the new test does not include correctly identifying President Bush as the person who has said "there ought to be limits to freedom," or, "If this were a dictatorship, it would be a heck of a lot easier, just so long as I'm the dictator." Presumably, he will not be writing the new test.

See what I mean? Already, the Cleveland Plain Dealer has injected political ideology into the discussion, and although the President himself will not author the test, the fact that it's being revamped underneath his command means that it might be more likely to reflect a Republican bent than a Democratic one. And there's no way on earth that the Bush-haters will agree with anything that the Republicans say correctly reflects "the values that make us one nation." Whose values? Who decides? And what about all those textbooks that are constantly re-interpreting history and reviling the "dead white males?" Do they guide the citizenship test? If not, why not? Why should native-born Americans learn one thing in high school while immigrants learn another thing in citizenship class?

And so, we're back to using the date of the signing of the Declaration of Independence as a test item. It's so much simpler.

But what should citizens know?...I knew a woman in college who refused to believe that someone named Hoover had ever been president. She identified Hoover with the FBI and the vacuum cleaner. Her major was education, and she did just fine.

I'm not even going to touch that one.

Maybe a civics test for the 21st century needs to go beyond traditional facts and history to deal with matters of real significance in American society. Here are a few suggested sample questions that might help:

If professional wrestling is fixed, how come they can still get hurt?

Sounds like as good a test as any.

The unbearable pressure of the ISEE

Here's an obsequiously sympathetic article about the students facing the hurdle of the Independent School Entrance Examination (ISEE):

For students aspiring to the nation's elite private schools, January means one thing: admissions tests, most notably the Independent School Entrance Examination, or ISEE. Think of it as the SAT for middle-schoolers, and for many kids, it comes with all the pressure and anxiety associated with the college boards.

With competitive schools such as Jesuit College Preparatory School of Dallas and The Hockaday School receiving applications from nearly twice as many students as they will admit, parents and kids say the key to getting in is a great ISEE score. The pressure is enough to make them pop a vein.

"One dad was there right up in his daughter's face, drilling her, telling her to focus," Elena said. "I felt so bad for her. How is it possible to handle that kind of pressure?"

Do we know for sure that the dad was harming his kid, rather than, you know, helping her to focus? If he was screaming, that's not good, but there's nothing wrong with helping kids prepare for exams. And "palpable anxiety" pretty much describes my entire high school experience, including tests, dances, and having to change into skimpy gym shorts every day (I wore mine over a pair of sweatpants). The fact that this test produces anxiety doesn't mean the test is necessarily a bad thing, if that's what we're supposed to conclude from this article.

Several Ursuline students described parents who were so nervous over the ISEE that they cried on test day. Others said they know of parents who danced around the mailbox when the scores arrived.

"The myth that the ISEE makes or breaks your application isn't true," said Jesuit admissions director Tim Host, who called parent fever over the ISEE "madness." Yet he conceded that "when it comes down to those final 10 spots and we have nothing else to differentiate kids with, then yes, a bad ISEE score could break them."

Yes, when all else is equal, the ISEE could be important. Some parents will cry, and some will dance. Sounds a lot like life to me.

The ISEE is a three-hour, multiple-choice test with a short essay, similar in format to the SAT. It covers verbal and quantitative reasoning that measures a child's capacity for learning. The reading comprehension and math portions of the test are designed to pinpoint a student's weaknesses in those subjects. And because the test is generally taken by high-achieving students, kids who ace other standardized tests are often surprised to see how they stack up against other bright kids.

"It is expected that children generally will score lower on the ISEE than other nationally normed tests," said Elizabeth Mangas, executive director of the ISEE.

That's one argument in favor of providing at least some test prep for your kid for the ISEE, even if it's just multiple pep talks that focus on the fact that this isn't an easy exam. It's meant to differentiate amongst the brighter kids, so it might seem intimidating.

The private school officials downplay the importance of test prep along with the importance of the ISEE, and that's not surprising. They're aware that testing fears are what generate $100-an-hour fees for coaches, when it sounds like a good ISEE may not balance out an otherwise-unimpressive transcript.

Toiling under the "looming" FCAT

The FCAT "looms" over Florida's students, according to the TCPalm website:

Although the 15 students [in the room] are not taking practice Florida Comprehensive Assessment Tests, in some ways the state's standardized test is always in the room. The children are together for two hours each day because they scored at the lowest of five levels on the FCAT reading section. They are not alone.

FCAT scores and increasing proficiency are constant concerns for administrators and teachers. That brings stress, but most agree: The material on the test is material students need to know...

Many teachers who support the test are quoted as saying that they don't teach to the test, but instead teach "a deep, rich curriculum [with] good instruction." But at one high school, most of the freshmen and sophomores have had to give up an elective to take an FCAT-focused research class. I can understand why this doesn't sit well with most teachers, but it's hard to understand why the basic FCAT skills aren't being conveyed in other classes.

The inside scoop of Harcourt Assessment

Here's an article in My San Antonio online that does a great job of describing how tests get produced and scored at the Harcourt Assessment organization:

In the second [HA] building behind the main lobby, certain corridors are off limits, and visitors are flanked by helpful, cautious guides. In there is the massive warehouse where millions of pieces of paper are tracked and sorted by radio frequency, with two sets of loading dock doors to segregate incoming from outgoing.

"In some of my essay questions, I would see 'If you're reading this stuff, it must suck to be you.'" Loxsom said. "I thought, 'You kind of nailed it there, kid.'"

Despite the anti-testing backlash, Harcourt is growing rapidly:

The San Antonio-based company, now a subsidiary of Dutch company Reed Elsevier after almost a century of name changes, mergers and acquisitions, basically doubles its staff of almost 1,300 during the peak scoring season.

The company also added 367 employees last year and plans to keep growing if it can find enough statisticians and psychometricians, who are specialists in the measurement of academic achievement. It wouldn't release figures on what it pays or what it earns but called 2003 "a successful year."

That "if" above is crucial. The number of tests required each year keeps growing, but that's not being met by an increasing number of psychometricians. Even the executive director of FairTest, who's never met a test he liked, is quoted as saying that one of the main causes of scoring errors is the understaffing of psychometricians at companies that are pressured to turn scores around quickly.

Harcourt has apparently started a fellowship program to "woo" more students into the field of psychometrics. ETS, LSAC, and ACT already have similar programs in place (full disclosure: my last two years of graduate school work were supported by an ETS fellowship). Here's a press release advertising the fellowship.

Note the names of the four doctoral fellows who spent the summer at Harcourt in 2003, learning the ropes. America is a mecca for would-be psychometricians from around the world (the Netherlands is a close second).

Fighting stereotype threat

Fascinating article in the NYT about raising test scores by fighting the pressure of negative stereotypes:

Girls and low-income minority students are more likely to improve their scores on standardized tests when they are taught ways to overcome the pressures associated with negative stereotypes, according to a new study of seventh graders.

The girls who were taught that intelligence developed over time scored significantly higher on the standardized math test than girls in the fourth group. Similarly, the minority and low-income students who were told that they could overcome challenges and achieve academic success scored significantly higher on the standardized reading test than students in the fourth group, the researchers found.

Hoorah. Emphasis mine. These types of findings highlight what I find so noxious about the "all-tests-are-biased," anti-testing crowd. It does no good whatsoever to scare students into believing that test scores are arbitrary or that high test scores are out of reach for black or Hispanic students. Those who claim that all standardized tests are by nature "racist" or "sexist" are guilty to contributing to a lack of confidence on the part of minority and female test takers. They perpetuate these negative stereotypes, which then have to be overcome before students can perform up to their full potential.

Avoiding the motivation of high test scores

And the articles insisting that testing is mutually exclusive from "real" education just keep on coming:

For instance, a new commission of the national government, business and education leaders wants to pay teachers based on how well their students do on standardized tests. That would mean that a teacher's pay would be partially based on student test-score gains. Another part of the pay formula would be based on well the entire school tests out with the remaining 50 percent based on evaluations of how well the teacher motivates students in the classroom....

We tend to worry, however, that the importance of education is being given short shrift under this formula. Teaching children to do well on a standardized test isn't the same as turning kids on to the value of education and opening the doors of knowledge to them.

Why not? Why is there any reason to assume that students who do well on tests haven't been "turned on" to the "value of education"? Why does testing have to be inconsistent with this? Why can't teachers "open the doors of knowledge" in ways that will allow children to demonstrate their understanding on exams?

Standardized tests measure educational concepts, skills, and facts. Students who do poorly on these test have not learned these concepts, skills, and facts from their teachers. It's legitimate to argue whether a particular test is measuring an appropriate set of concepts; it's ludicrous to claim that all tests are by nature inconsistent with actual learning.

This author's statement isn't just insulting to test developers, but also to students who do well on tests, because by this writer's logic, we cannot assume that someone who makes a high score on a test has in fact learned anything. I wager most high scorers will beg to differ.

Some teachers point out to parents that lessons will be geared to state PSSA tests when the school year starts, and this creates a different kind of formula under which all apparently are under the gun to teach the same skills at the same time.

And this is a problem because?

We bring this up because of concerns that some ingenuity in teaching will disappear as the emphasis builds on teaching children specific skills to answer test questions on a timetable. This approach could make learning more disjointed and have less correlation to overall studies.

Say what? How is following a sensible timetable to teach a specific set of skills "disjointed"? And no one is telling teachers how to teach, only what to teach. A teacher who has "ingenious" ideas about whether kids need to learn to read or do long-division is not her students any favors.

Trust me - truly ingenious teachers produce kids who score well on tests because they thoroughly understand the material.

Forcing everyone into a set pattern of lessons may not be the best way to get to the objective of excellence. Teachers as well as students need motivation. Money is a motivator, but there has to be more than money to experience the greatest rewards of a profession.

I agree, but thanks to the negative barrage of anti-testing comments like these, teachers aren't taking advantage of an excellent non-monetary motivation - test scores. Articles like this one are an attempt to warn teachers away from monitoring their students' test scores. This article is basically telling teachers that they cannot help their students do well on tests and "genuinely" educate them at the same time. Don't you think that if teachers understood the link between test scores and achievement, and were not told that tests are unfair and inappropriate, they'd take more pride in their students who perform well?

Your concern is touching

A "Very Concerned Student" has written to the Jupiter Courier about the FCAT requirements. I'll go easy on her - but I'm not letting her off the hook entirely:

Dear Governor Bush,

I know that much controversy exists over the benefits of the FCAT. I understand all the rules and regulations of the FCAT. Even if a student sustains a 4.0 grade-point average but fails the FCAT, he or she will only get a high school certificate of completion and, as a result, will not qualify to attend most colleges.

I am not the only person who can see that there is definitely something wrong here.

Yes, dear, there is, but it's not what you think. First, the FCAT measures 10th-grade skills, and students get multiple attempts. If a senior with a 4.0 can't pass it, why do you assume the test must be wrong? Wouldn't it be equally likely that rampant grade inflation and dumbed-down classes are to blame? Why shouldn't a straight-A senior be able to pass a sophomore's exam?

Let's extend the logic. Say the test shows that a 12th-grader with good grades is performing at 10th-grade level. You believe this student should get a real diploma, which will "qualify" them to attend college. But they won't really be qualified, because they're still only doing 10th-grade-level work. They have a piece of paper that says they ARE qualified, but that piece of paper in and of itself will not MAKE them qualified.

I strongly maintain that all students be treated as individuals and be given the best possible educational experience. Standardized tests fall short of this goal, especially when advancing to the next grade, or going on to higher education, is absolutely dependent on passing such exams.

I agree entirely with your first sentence. What you don't seem to know - because, in all likelihood, journalists and teachers have never mentioned it - is that students who receive "the best possible educational experience" will pass any exit exam in the country, and possibly the world. "Going on to higher education" is not "dependent on passing such exams," it is dependent on the student learning the material necessary to pass the exam. The exam is a proxy for learning. Student who know the material will pass. If the student does not know the material, then moving on to college may not do them much good.

Of course, tests are not 100% reliable, and we don't know, without examining the situation further, how valid the FCAT is in this situation. But when big whomping numbers of 12th-graders fail 10th-grade exams, something is indeed wrong, and it's just as likely to be wrong on the class end as on the exam end.

Additionally, I have read that thousands of third-graders were held back this year, not allowed to advance to the fourth grade because they did not pass the FCAT. How is this exam a benefit in this situation? The individual student is now stigmatized because he was "left back."

No, he's stigmatized because he can't read. Next year, he may very well have fallen even further behind. While there are issues with the policy of retaining such huge numbers of third-graders, the question we should be asking is not, "How is this exam a benefit in this situation?" but instead, "Why are thousands of little Floridians so far behind already?"

I believe the schools are spending a good portion of the year preparing and rehearsing for this test. It is so excessive, in fact, that not enough time is spent on subject teaching. The importance for the tests is so overstated that many children are too stressed out and anxious. I understand that some anxiety is beneficial for test taking. However, when there are children getting sick due to too much stress, then it becomes a problem.

Sing it with me, folks, if you know the words - What good does "subject teaching" do for kids who can't read? What good is 12th-grade History for a kid who failed a 10th-grade Reading exam? And if the kids are anxious, who helped make them that way? Perhaps the educators and naysayers who are always claiming the tests are unfair, biased, racist, etc?

You know, it would make me sick, too, if I constantly had to take tests for which my teacher failed to prepare me. One way to fix that is to change the teaching method. "Teaching to the test" is a good thing if it's a good test measuring necessary skills. Funny, but few people complain about those driving instructors who teach in alignment with the driver's license exam.

Even the media points out problems with this system.

That's too priceless; I can't even touch it.

You must appreciate that there are major issues resulting from this test. I have yet to learn that the FCAT has, in some way, any way, benefited a child's educational experience.

The FCAT in and of itself is not the change; it's a measure of the change. Those of you curious about the change that it's measuring can click here. More evidence of change will come down the road, with future NAEP scores, SAT scores, and dropout rates.

Update: The always-interesting Winston's Diary has the inside scoop on the syllabus for a community college composition class:

Yes, I taught a composition class in which I was required to begin with parts of speech, move on to writing basic and then complex sentences, move to the paragraph, and then finish--if possible--with a 3-5 paragraph essay. This was done in an 18 week semester, so I spent about three weeks on parts of speech, and several students failed quiz after quiz on things as simple as nouns and verbs, subjects and predicates. I actually resorted to Schoolhouse Rock cartoons to try and get them to memorize the songs and, hopefully, the content. A failed effort, though; by the fourth week of class, we had gone from 30 students to 14. Most of the students who dropped the course were performing in the 10th to the 40th percentile.

All of my students were high school graduates. I'm not saying they graduated with a 4.0, but they were in possession of diplomas. I know this because we discussed it. They were in no way prepared to attend college, so the college was forced to offer not only four semesters of remedial English but several semesters of remedial math. Many students were enrolled in both remedial English and math, and were going to be in junior college for at least an extra year trying to get themselves up to the junior college level.

Depressing, isn't it?

Bashing the tests, missing the point

This article in The Midweek News (DeKalb, IL) is an amalgamation of just about every bad testing-related education idea that exists, along with just about every bad testing-related journalistic approach possible. I'll list a few for your amusement; see if you can find more!

What a couple of decades ago was considered “teaching to the test” is now called “aligning the curriculum with state standards.”

Well, at least this bit of sophistry reveals the truth behind those educators who oppose "teaching to the test," which is that they don't want the material they teach to have to conform to any objective standards.

The carrot is continued federal funding. The stick is an escalating series of sanctions, including a cut-off of funds and possibly a closure or the imposition of private management on the school. According to critics, the only thing that counts is the number of students who score above the basic level.

Can those critics provide a reason why federal or state governments should not be concerned with the relatively large percentages of students who score at or below basic levels on many standardized exams? These critics seem to have confused "valuing all students" with "being complacent with 1/5th of all students performing below the most basic levels of core skills."

Sycamore School Superintendent Robert Hammon frequently comments to his Board of Education on the difficulties with NCLB, and he noted, for the first thing, the law is 1,000 pages long “and that doesn’t include the rules.”

He explained, “The original intent was admirable, but (the actual effect) is in the details and it obviously has created a whole new level of bureaucracy. Also, it certainly has removed control from local educators (and taken it) to the national level.

Would this have been necessary if the sentiment in Washington was that local educators were doing their jobs right? Obviously some of them were, but large numbers of them weren't.

“If we’re going to play the game and be judged by it, we will have to focus on nothing more than the tests,” he said. “The question will then become, ‘How is what I’m about to do going to improve the test scores?’” Hammon said.

Funny, many schools that have revolutionized their communities with "unteachable" kids rely a great deal on testing. And they ask themselves this very same question; unlike Hammon, they haven't already decided that all possible means of genuine education are unrelated to test scores.

Underscoring his point, the journal of the National Board on Educational Testing and Public Policy headlined, “The higher the stakes, the more teaching to the test.”

In other words, he's noticed the stunningly-obvious (to anyone else) concept that when schools claim skills A, B, and C are important, then teachers will focus on teaching A, B, and C. Those skills can be as meaningful and broadly-defined as possible, but if schools do not have to teach them, some schools will not teach them.

NAEP scores make this rather obvious. Take a look at recent Reading score reports, for example. Reading is the fundamental task that should be imparted within any type of educational system in the US, and yet look at the whopping percentages of fourth-graders who are already performing Below Basic in Reading. Sixty-nine percent of DC's fourth-graders are at this lowest level on NAEP.

If those sixty-nine percent were to spend one year with a teacher who was obsessed with nothing but "teaching to" a reading test, they'd be better off than they are now.

Hammon said other educational goals such as music and the fine arts, teaching students to be critical thinkers and to be good, solid citizens involved in community service will fall by the wayside.

These educational goals will never fall by the wayside. What's more, ensuring that all children have basic skills in reading, writing, and math will help ensure that these kids can make better use of their "critical thinking" skills. If they can't read, then what the heck are they "thinking critically" about now?

[One educator] posed a hypothetical situation. “You are helping a bilingual, special education student from a low-income family to raise his or her score, which is currently 50. The passing score is 200, and the child’s score increases from 50 to 199. A score of 199 is a failure.”

But that student is much better off than they were when they were at 50. And would they have tried hard enough to reach 199 if the goal of 200 was not there?

So, what journalists cliches are in evidence? Quotes from testing critics, but none from testing supporters, developers, or psychometricians. Brief mention of the heavy support that NCLB receives from its penultimate consumers - parents - without any discussion of why educators are so in opposition to something that parents favor. Unsupported claims of how "learning" can't be represented by tests are presented as though they are uncontested facts. Horror stories about how gifted student funding is shrinking while barely literate kids get more time and attention, but no discussion of why there are so many barely literate kids in schools these days. And so on.

My favorite graf?

Assessment specialist Fames Popham predicted, “We will witness a growing clamor from citizens who just don’t believe their local schools are rotten....As soon as more than half the schools nationwide are labeled ineffective, then the requirements will be modified.” He recommended a public information campaign to educate citizens as to the problems with NCLB.

Got news for you, Popham, the numbers of parents who DO think schools are rotten is growing much faster than those who are complacent about it. Hence the rise in charter schools and homeschooling. What's more, parents are too smart to believe that most anti-testing educators are basing their opposition on anything other than a desire to save their own hides.

That much said, there are problems with NCLB, testing is not the same thing as education reform, and having to meet 100% of the goals is unrealistic. I've never denied that. But what the educational community doesn't seem to understand is that desperate measures tend to follow desperate times, and many, many parents of schoolchildren have, over the last 20 years, considered the state of public education to be about as desperate as it can get. Any educator who doesn't acknowledge this is not going to be taken seriously when bashing the NCLB Act.

Don't like the NCLB? Come up with something parents like better, or get the heck out of the way.

From bad standards to a worse test

Both the Catholic School Blogger and ReformK12 recount the "horror story" of changes to the Algebra I curriculum, as described by a student at Montgomery Blair high school in Maryland.

Bottom line: The Maryland State Dept. of Education wrote a new set of standards that "take the algebra out of algebra," according to one critic. The statewide test (the High School Assessment, or HSA) was then re-written to match the new standards. Schools such as Montgomery Blair re-wrote the curriculum to match the tests. And the response of teacher at the Blair school (which, if the online newspaper is any indication, is a top-notch school) is overwhelmingly negative:

Concerned about her students’ performance, Pre-Calculus teacher Julie Greenberg asked her colleagues via email about the competency of their current students. Forty teachers responded, 29 of whom indicated that their students were "less competent" than those of their earliest teaching experience. The majority of those 29 cited basic algebra skills as the root of their students’ deficiencies...

Data analysis represents one of the seven Algebra I units, and that unit takes less time than most of the other units, said MCPS Mathematics Instructional Specialist Lauren Duff. However, MCPS documentation for the Algebra I curriculum recommends spending six weeks on the "Data Analysis and Probability" unit, more time than on any of the remaining six units in the curriculum.

The battle lines in mathematics education appear to be drawn, with the National Counsel of Teachers of Mathematics (who guided the development of the new math standards) on one side, and those who believe in teaching fundamental skills on the other side.

Many believe that this problem is not only limited to Algebra I and other courses with HSA requirements, but that it is a symptom of a larger movement in math instruction that spans K-12.

Walstein cited the increased use of calculators in MCPS curriculum as an example of this problem. "If students can just punch things into a calculator, and it spits out the answer, that’s not math. They’re not learning anything," said Walstein. While Walstein believes that calculators should be used in certain circumstances, they are not a substitute for understanding the material.

Mathematically Correct has a good summary of a set of sound mathematical standards that go from kindergarten through Geometry. The website also provides a link to a scathing critique of the NCTM standards. Key comments (with respect to revisions to the NCTM standards in 2000):

The NCTM has toned down the constructivist language, but they still stress content-independent "process skills" and student-centered "discovery learning". Similar to the NCTM Standards, PSSM emphasizes manipulatives, calculator skills, student-invented methods, and simple-case methods.

The NCTM says they want to maximize "understanding", but they still fail to recognize that specific math content must first be stored in the brain as a necessary precondition for understanding to occur. Although rarely the preferred method, intentional memorization is sometimes the most efficient approach. The first objective is to get it into the brain! Then newly remembered math knowledge can be connected to previously remembered math knowledge and understanding becomes possible. You have to "know math" before you can "understand math", "do math", or "solve math problems."

Also mentioned is an article by Berkeley Professor of Mathematics H. Wu about the "false dichotomy" in mathematics education:

Education seems to be plagued by false dichotomies. Until recently, when research and common sense gained the upper hand, the debate over how to teach beginning reading was characterized by many as "phonics vs. meaning." It turns out that, rather than a dichotomy, there is an inseparable connection between decoding—what one might call the skills part of reading—and comprehension. Fluent decoding, which for most children is best ensured by the direct and systematic teaching of phonics and lots of practice reading, is an indispensable condition of comprehension. -Wu, Page 1

"Facts vs. higher order thinking" is another example of a false choice that we often encounter these days, as if thinking of any sort—high or low—could exist outside of content knowledge. In mathematics education, this debate takes the form of “basic skills or conceptual understanding.” This bogus dichotomy would seem to arise from a common misconception of mathematics held by a segment of the public and the education community: that the demand for precision and fluency in the execution of basic skills in school mathematics runs counter to the acquisition of conceptual understanding. The truth is that in mathematics, skills and understanding are completely intertwined. In most cases, the precision and fluency in the execution of the skills are the requisite vehicles to convey the conceptual understanding. There is not 'conceptual understanding' and 'problem-solving skill' on the one hand and 'basic skills' on the other. Nor can one acquire the former without the latter. - Wu, Page 1

And speaking of "false dichotomies," there are, unsurprisingly, comments following the original SilverChips article that make the following assumption: On one side is genuine learning and a set of meaningful standards; on the other side are bad standards and all high-stakes tests.

Although the tests in this Maryland case are compounding the problem, the tests aren't at the heart of the problem. Yes, a test based on poor standards will not help education; in fact, it will probably make matters worse. But this is not evidence that high-stakes testing is necessarily flawed. The problem here is the set of standards, not the test.

The Praxis debate in Virginia

Apparently, Virginia has the toughest passing standard on the Praxis I in the nation. Until now, Virginia's teachers had to meet that standard - but if these proposals currently before VA's Board of Education are accepted, that won't be the case any longer:

The state superintendent of public instruction is recommending that the board, which sets education policy for Virginia, accept college-entrance SAT scores in lieu of passing scores on the Praxis I, a timed, standardized, high-school-level test of basic reading, writing and math. Another recommendation is to establish an appeals process for teachers who narrowly miss passing the Praxis I.

The appeals process makes sense, especially if a teacher can present evidence showing that the reliability of the Praxis I is such that a teacher with the required ability can nonetheless sometimes fail by one point. Unless the test is perfectly reliable, this can happen.

The part about the SAT doesn't make sense. The SAT is a standardized test that measures whether students have the math and verbal skills necessary for college. The Praxis I is, as described in this article, a test of "high-school-level test of basic reading, writing and math." If, after two or four years in an education program, a teacher can't pass the Praxis I, what reason is there to suppose that their previous SAT scores would be stellar?

According to this report, while the average SAT scores for all candidates in 1997 were 505 on the verbal section and 511 on the math section, the average verbal and math scores for candidates seeking a degree in education were 485 and 479, respectively. It is doubtful that this trend has changed; thus, there's no reason to conclude that education majors are likely to be performing above average on the SAT.

The report also concludes:

The answer is essentially the same for both Praxis I and Praxis II: At the low passing score, passing rate gaps between ethnic groups drop, but so do the average SAT and ACT scores of the people who pass. High passing scores increase ethnic gaps, but significantly increase SAT and ACT scores as well. For example, at the low passing score for Praxis I, 67 percent of the African American candidates would pass, as opposed to just 17 percent at the high score threshold. The corresponding percentages are 95 percent and 52 percent for white candidates. These figures are especially worrisome given the current dearth of minority teachers.

However, if every state were to go with the high passing scores on Praxis I, prospective teachers' average math and verbal SAT scores would each climb more than 40 points higher than with the low passing scores. Policymakers thus face a vexing decision: They must weigh the relative merits of academic ability and teacher supply, particularly with respect to the supply of minority teaching candidates.

In other words, those teachers who can pass Virginia's Praxis I standards are likely to have higher SAT scores than average, but there's no reason to assume that those who can't pass the Praxis will have a high enough SAT or ACT score to assure the state of their competency. The one issue here might be that the Praxis is on computer while the SAT is not, but it's hard to believe that the mode of testing is so problematic that it causes high SAT scorers to do poorly on the Praxis.

If this proposal were accepted, what SAT score would be necessary for a prospective teacher to avoid the requirement for the Praxis I? The acceptable SAT scores in other states range from 1000 to 1100, or from average to slightly above average.

The Praxis I score scale, for those of you unfamiliar with the test, provides separate verbal, math, and writing scores. The current Virginia standard is a 178 on both math and verbal, and 176 on writing, for a combined scores of 532. The lowest standards for other states seem to be set around 170. According to this ETS report, the top score on Praxis varies from year to year, which makes this a bit hard to understand. The minimum combined total score is listed as 780, which I assume means 260 on each section. The only report I could find listing a Praxis national average was this, which reports the average as 172 for math and reading, and 171 for writing.

Thus, those teachers who fail to meet Virginia's standards by one or two points will still be above the national average on Praxis I, and thus may have an SAT score above the national average as well. However, their Praxis I scores would still be within an acceptable margin of error for the Praxis, and so that information, rather than their SAT scores, should be what gets them their teaching certificates.

The other possible criteria for appeals look more appealing than the SAT option:

* Prospective teachers pass two of the Praxis I subtests, and on the third score within a specified margin of error that would vary with each test. (This covers the reliability issue I mentioned above.)

* They have successfully taught in Virginia for at least one year. ("Success" would need to be well-defined.)

* They receive a recommendation from their division’s superintendent, or their private school director.

* They fail the test three times, with tutoring in between. (Does this just mean that three failures are required before one can appeal?)

Also suggested are examining classroom skills and student rapport, which would be helpful for a teacher on the brink but not one far away from the cutscore. "Rapport" should not be allowed to compensate for lack of a firm understanding of basic math and reading skills.

Is the SAT dead?

Psychology Today ponders this question in their online magazine:

The SAT has long been a three-letter word for aspiring college students—an abbreviation that stands for fear. It has become the single most important arbiter of university admissions, the magic number that seems to predict the future for nervous teens. But the tide may be turning, at least among elite private schools. The latest nail in the coffin: Sarah Lawrence College, a prestigious liberal arts school in Bronxville, New York, recently announced that beginning with the high school graduating class of 2005, applicants will no longer be required to submit standardized test scores.

The decision was intended “to reflect our belief that standardized testing is not effective in evaluating a student’s ability to succeed in a writing-based curriculum such as ours,” the college’s dean of admissions said in a press release. The statement also fretted over the growing inequity between test-takers who can afford preparation courses (which can cost up to $900) and those who cannot.

Not a bad beginning, but I already have several comments:

(1) Is the SAT score really the "single most important" factor in the admissions decision-making process? Almost certainly not, especially for lower-tier schools, and not in this age of affirmative action and "comprehensive review."

(2) Is the decision by Sarah Lawrence really the sign of a trend against the SAT? After all, the college isn't opposing the SAT on principle; it's perfectly legitimate for a college to base grades on writing skills and then to decide that the SAT isn't a good predictor of those grades, for that college. There's no reason to suggest that this "trend" will catch on with colleges that place emphasis on other types of performance.

(3) Tuition at Sarah Lawrence College is over $28,000 a year. And we're supposed to believe that they're really worried about applicants who can't scratch up a measly grand for test prep? That sum isn't even necessary - a prospective student can prepare for the SAT using 10 practice exams for $19.95. Come on. Sarah Lawrence is more than entitled to decide that the SAT is not right for their students, but not on the grounds of expense.

Anyway, let us continue...

The SAT is the nation’s oldest and most widely used college admissions test. It is also big business: The College Board earns millions each year in revenue from the exams.

And your point is? Is the College Board supposed to give the test for free?

But its critics have gained momentum in recent years. They say that the SAT is a poor predictor of women’s, non-native English speakers and older students’ academic performance in college. Whites outscore African Americans on average by 206 points.

"They say"? Come on, PT, you have to do better than that. "They" can say anything; whether they back it up with data is a different story. What's more, the utility of the SAT can vary from school to school, and it's inaccurate to claim that, for example, the SAT does not predict well for women across the board. The mention of White scores vs. African American scores is a non-sequitur here, regardless of its accuracy. The score gap is related to differential impact, but not to bias; nor is the gap any indication of whether the scores for both groups are differentially useful.

When the article does get around to quoting someone, that person is, unsurprisingly, a spokesman for an anti-testing organization. The National Center for Fair & Open Testing hastens to mention that "700 colleges and universities" no longer require the SAT, but the reporter should have wondered how many of those were recent changes, and should have noted that those 700 comprise less than 17% of the 4182 private and public two- or four-year institutions in the US.

The only other person quoted is a Yale psychology professors who thinks the SAT isn't adequate for predicting college performance. Neither testing critic provides studies to back up his claims. And the conclusion is a load of anti-wealth and multiple intelligence blather:

Sternberg says that the SAT tests well for memory and analytical skills. “Clearly, these are important,” he says. “But in life, and in college, you need more than analytical skills.” The test is not so good at gauging creative skills and practical skills, which means that some kids may not be fairly ranked by the exam. “A kid from a challenging environment might have better-developed creative or practical skills, whereas a kid who grew up in Scarsdale [may have] had the luxury of developing analytical skills,” says Sternberg. His recommendation: Admissions officers should learn how to recognize a diverse set of skills or risk rejecting talented applicants.

Does Sternberg really believe that admissions officers don't already do this? No university in the US relies solely on SAT score for admissions, and College Board policy is that the SAT should not be used as the sole method of selecting students. What's more, the reference to "Scarsdale" is ridiculous. Depending on the university and the program, "creative skills" may well count for nothing when compared to the need for analytical skills. Sternberg might think that it is a "luxury" for a student to have had the opportunity to develop analytically (if true, that's an indictment of the K-12 public school system, not a criticism of the tests), but it's just plain silly to conclude from this statement alone that analytical skills should be downweighted in the college admission process.

C'mon, Psychology Today. You didn't even make a half-hearted stab at even-handedness. One measly quote from someone at the College Board - or from any psychometrician - would have gone far towards making this article something other than a one-sided essay about the potential unfairness of standardized tests. For example, I could have pointed out that the artificial restriction of range of the SAT scores of admitted students could be depressing the SAT-college grade correlation. Also, it's a crime to write an SAT article and not quote expert Wayne Camara, who has published research supporting, among other things, the usefulness of the SAT in predicting college grades for women. He also notes that the claims that huge numbers of colleges are dropping the tests may be greatly inflated, and says that between 92% and 96% percent of four-year institutions require an admissions test.

There ARE valid reasons for a university not to require the SAT, so I don't have a problem with the idea of pondering the utility of the SAT. It's just a shame to see an article waste the reader's time with the unsupported and one-sided arguments made here. What's more, I'm not even sure what conclusion the reader is supposed to draw from this article.

For example, if the SAT suddenly went out of favor for all universities, the College Board would simply develop another test more to the liking of the administrators (in fact, that's how the new SAT revisions came about). Five years from now, the College Board might pop up with a new writing exam that better predicts student performance at colleges like Sarah Lawrence. If the writer was hoping this article would make readers root for the bankruptcy of the College Board, she's got it all wrong.

Setting state-level standards

Great article in the NYTimes on December 31st on the problems with assessing national student proficiency when state test difficulties vary so much (thanks, Devoted Reader Susan!):

The community around South Charlotte Middle School is one of the richest in North Carolina, and the school boasts the kind of test scores that seem to go hand in hand with wealth. Last year, more than 95 percent of its students passed both the state reading and mathematics tests.

The difference? Fort Mill Middle School is in South Carolina.

And South Carolina has one of the harder fifth-grade exams; it's estimated that 75% of fifth-graders nationwide would fail it. Two recent studies reached the same conclusion: "Across the country, there is no agreement on how much students need to know to be considered proficient.":

The divergent standards also have ramifications under the federal education law, passed in 2001. Schools deemed failures eventually face stern consequences, including loss of students and reorganization. And in some states with high standards there could be lots of failing schools. In other states with low standards, schools with equally poor performance could be left alone.

States are given the freedom to set standards based on the needs of their children and their communities. The results are, not surprisingly, wildly divergent from state to state:

The Northwest Evaluation study was based on scores of students in 14 states who took both the state proficiency test and one of the organization's own tests. The group claims a high degree of reliability in its estimates, even though it may give its tests in just one school district, because they are based on scores of hundreds of students taking both tests.

The study by Achieve, a nonprofit group that promotes high education standards, compared the number of students the states had declared proficient under their No Child Left Behind testing structure with the number at the "proficient" level on the National Assessment of Educational Progress, a test given in all 50 states. It showed Louisiana, South Carolina and Wyoming setting some of the highest bars, with Texas, North Carolina and Mississippi setting some of the lowest.

In South Carolina, 75% of schools were listed as failing to make progress, compared to 3% in Alabama. Supposedly parents are aware of the difference in test difficulty and use that when judging schools, but it's hard to believe that some won't be swayed by the differences in the numbers. As long as states are allowed to set their own standard, I agree that a national clearinghouse or database is necessary so that parents have an idea of how their state's test compares to others.

So, why's that SC exam so challenging? Let's take a look. Here are the Grade 3 - 8 sample items for English Language Arts. For Grade 5, the 318-word passage concerns the possible extinction of manatees due to human interference. Leaving aside the enviro-ideology here, I'd say the item does indeed look challenging, but not confusing. Students have to draw a conclusion, figure out the meaning of a perhaps-unknown word, analyze text details, figure out what website they'd access to get more information, and write a summary - in complete sentences - relating to the claim that manatees may become extinct.

Nothing confusing about this, as I said, but nothing's a giveaway either. The standards for Grade 5 in English Language Arts are here.

Okay, so what's the difference between this and what Alabama's fifth-graders see, if only 3% of Alabama schools are listed as failing? Alabama's Department of Education page is rather confusing; I finally had to resort to Google to find any sample assessment items. This site has Grade 5 Writing items.

No reading passages are mentioned (but that doesn't mean there aren't any). Also, all the items are multiple-choice and seem staggeringly easy. There aren't any "answer in complete sentences" type items listed. If anyone out there has more information on Alabama's exam or standards, let me know.

One retired teacher leads the FCAT pack

A retired teacher's attitude towards the FCAT: If you can't beat 'em, join 'em.

James E. Feazell Sr. doesn't want to fight the FCAT. As long as the standardized test is required, he's not going to challenge the rationale behind it. He would rather help the kids who struggle to pass the exam... As his retirement approached this past July, he wondered how he could help black students he knew who had failed the FCAT.

Others accuse the state of discriminating against minority students, but Feazell felt that complaining wasn't going to change anything. His mission became to help improve their test scores.

Bravo. Feazell feels that he received a great amount of help his entire life, from the Good Lord on down, and so he wants to give something back to his community in turn.

Feazell formed a partnership with various Pinellas County schools to create the "Bridging the Achievement Gap" program. The schools sent letters to 229 parents whose children had failed the FCAT, telling them about free tutoring in their neighborhood. Four seniors and five juniors came for tutoring three times a week in September. After two and a half weeks of tutoring, they took the FCAT. Three of the four seniors and all five juniors passed.

Good grief, why did he attract only nine students out of 229? For free tutoring? Did the others feel they didn't need it? Or were they too busy complaining instead of studying?

Now, about 45 students from Largo, Seminole, Osceola and Pinellas Park high schools attend tutoring sessions at Shiloh Missionary Baptist Church and Young Life Center in Ridgecrest. With six Largo High School teachers and 10 student tutors, each group is capped at four students.

"I think this is going to be a model for other communities, other schools," said Largo High principal Barbara Thornton, who sits on the program's advisory board. "He's dedicating his life to it. When you create something like this, somebody has to drive it. It has to be somebody with this vision - and he has it."

Again, bravo.

Two high scorers diss the exams

Now, here's a twist. Two students from Monroe, NJ, aced the PSAT - and then told a reporter that intelligence doesn't matter on these exams, only practice:

According to Yushen Qian and Robert Ngenzi, a student's intelligence should not be judged by how well they perform on a standardized test.

"Nothing beats effort and hard work. You can be the smartest person and not put in any effort and bomb it," Yushen said of taking a standardized test.

Yushen thus makes a logical fallacy that is often made by testing opponents, and that is:

If A is intelligent yet can flunk an aptitude test due to lack of practice (i.e., a complete unfamiliarity with the exam), then B can be unintelligent and make a high score solely due to practice.

The first part might not be true; the second part definitely isn't.

There's no reason to assume that a student who is smart, and accomplished, could bomb the SAT merely due to lack of practice. Few students would be completely unfamiliar with the format, and, barring emotional upsets, most students would be able to concentrate for the few hours of the test administration.

A smart student who doesn't get prepare might not get as high a score as they otherwise would have, but that's not the same thing as saying that a smart student will "bomb" the SAT if unprepared.

On the other hand, a student who genuinely does not understand the material will not score highly on the test, preparation or no. If practice were really all that mattered, then it would be a cinch to prepare 8-year-olds to pass the SAT. It doesn't work that way. There has yet to be widely-accepted research supporting the existence of "test prep" methods that substantially increase scores for everyone. The only groups that claim this research exists want hundreds (or even thousands) of dollars for it.

Practice will help a student do as well as they possibly can on exams like the PSAT and SAT, because practice will familiarize a student with the exam format and teach him to pace himself. Both Yushen and Robert are in advanced-placement classes and participate in an extra-curricular competitive math and science group. It's not surprising that they were able to make perfect scores with intensive practice. What IS surprising is that no one else seems to have pointed out to them that this was due mainly to their high intelligence (supported by factors other than the test), and that it's folly to claim that students of lower intelligence and education could have scored perfectly with as much practice.

Robert even claims that "The SAT only tests how well you can take a standardized test." Really? Then I assume that when he and Yushen took their very first PSAT, they both made rock-bottom scores of 20. No? You doubt that's so? Then what was that test measuring if they did better than a 20 before all that practice?

I'd be willing to bet anyone $100 that both Robert and Yushen scored above average before any practice whatsoever. (I find it very interesting that the reporter apparently didn't ask about their score gains, and neither kid mentioned them.) I'd also like to know why both of these smart guys seem so unwilling to admit that their intelligence may have played any part whatsoever in their test scores.

More fretting about the Head Start exams

You know, I don't mind articles being written about the new Head Start exams, and I don't mind these articles being critical (as long as they're even-handed).

But does every reporter have to begin their Head Start article with a tale of a four-year-old's test anxiety? The New York Times used this as a lead in October, and the Portland Tribune did the same thing in December (albeit in a toned-down fashion). Now we have the SacBee weighing in with this oh-so-neutral opening:

Edward, with eyelashes nearly as long as the brush he was using to stroke blue paint onto a white paper plate, is usually a happy boy -- if a bit sensitive. When he was taken aside in November for a one-on-one test with a teacher, the 4-year-old's smile turned upside down.

"The whole time he was crying," said Karin Ramirez, site director of the Watt and E Head Start center. In between sobs, he wailed that he wanted to go back to class despite Ramirez's bribe of a SpongeBob sticker.

Sigh. I suppose I'm an ogre, then, for wanting to support the government's desire for accountability in the Head Start program?

"Head Start needs to be accountable," said Sharon Neese, manager of SETA Head Start in Sacramento. "We receive a tremendous amount of money. But an unresearched, thrown-together test is not a way to find out if Head Start is working."

What's the evidence that the test was "thrown-together"? And how can the tests be researched when Head Start managers don't want the feds to collect any data with it? For the test to be researched, four-year-olds have to take it.

...Horn, a child psychologist, said President Bush is trying to "manage by results," meaning whenever possible it is imperative to "measure what our good intentions are producing." Horn said the purpose is to identify local programs where more teacher training is needed. Federal administrators suspect some local programs are doing great and some not so well.

Probably a wise suspicion. The test required 18 months for development (that's "thrown-together"?), and the concept of measuring Head Start kids did not start with the Bush administration - only the standardized assessment format(essential for comparing federally-funded programs) and the federal reporting requirements (ditto) were recently added.

The exam was field-tested, and a 15-20 minute battery of questions was devised for the first go-round this fall. The results will serve as a base line. At the end of the Head Start academic year in May or June, the test will be repeated.

Okay, so not even the claim that the exam has not been "researched" is valid. Sigh, again.

Head Start's goal is to help economically disadvantaged children begin kindergarten on a level playing field with their more well-to-do peers.

Local Head Start leaders say parents of low-income children generally don't have as large a vocabulary as middle-class parents. So it is unrealistic, they say, to expect Head Start children to match their middle-class counterparts.

But, if the goal of Head Start is to put these low-income kids on a level playing field, then isn't it reasonable to measure whether Head Start kids learn a lot of that vocabulary? I mean, if the rationale is that their parental influence prevents them from learning anything, then why fund a program that is going to be ineffective?

Horn dismisses the notion: "What is astounding to me is that some who claim to be advocates say this is too high a goal. The president says that is nonsense. He says 'No Child Left Behind' does not mean just no rich child left behind. That is what the president is talking about when he speaks of the 'soft bigotry of low expectations.' "

He also said that worries about stressing young children are baseless: "My experience is that most 4-and 5-year-olds love to show off what they know."

In the right environment, I think that's as likely as the behavior of the otherwise-happy kid who becomes hysterical when confronted with a test item. So why not give us examples of both types of children? Or should we do away with anything that ever makes a four-year-old cry? This would include vaccinations and refusals to buy giant Barbie playhouses, I think.

High schools in SC win SAT cash

One high school in Columbia, SC, is one of 12 in the state to win cash for substantially improving SAT scores:

A.C. Flora High School has an extra $10,000 thanks to the student body’s stellar performance on the SAT...

“It’s made the kids more conscious of the importance of the test,” said Adams, Flora’s SAT team coach. “The publicity has been quite helpful.”

The various opinions on testing

Reporter Teresa Hoffman of the Ralston Recorder (NE) notes what readers of this blog have known all along - "Opinions vary on tests".

Some parents just want more of the testing info:

Kriss Kriglstein, a parent of a Blumfield sixth-grader, said she doesn't mind her son taking standardized tests, but said she would like to see more information on those tests given to parents and teachers.

Though she sees a need for the tests, Kriglstein said she thinks it's more important to look at the results of an individual child or even a classroom.

Some parents feel testing is too expensive and not well-explained to the consumers:

Janis Dwyer, who has a daughter in seventh grade at Ralston Middle School, said she has many concerns about standardized testing.

"Percentiles are not the same as percentage grade," she said. "A 50th percentile is average and the average of all numbers should be in the 50th percentile range to find a bell curve, which is the goal of standardized testing."

Unfortunately for Ms. Dwyer, who is a former teacher herself, she illustrates her point well by giving confusing and at least partially inaccurate definitions here.

The 50th percentile is not the average or mean score; it's the median, the point at which 50% of test takers score above and 50% below. The reason it's important to make that distinction is shown in the following example.

We give a test that has a score range of 1 to 100 to two groups of 11 examinees each.

* Group A contains scores of 30, 40, 50, 60, 60, 60, 70, 70, 70, 80, and 90.
* Group B contains scores of 60, 60, 60, 60, 60, 60, 90, 95, 95, 100, and 100.

The median, or 50%, of each group is the same - 60. But the mean for Group A is 61.81, while the mean for Group B is 76.36. What's more, the kid who's at the 50th percentile in Group A is indeed near the middle of his class, while a kid who scores at the 50th percentile in Group B, which is bimodal, is a member of that part of the class which is just not "getting it."

I assume that Ms. Dwyer meant to say that, if the distribution of examinees is a bell-curve, the 50th percentile will be (as it is in Group A), close to the average. However, if a test is criterion-referenced, there's no reason to assume that a particular cohort of students will make up a bell-shaped curve, or anything close to it. There's no reason why, on a criterion-referenced test, to assume ahead of time that a class would look more like Group A than Group B.

Thus, for any parent interested in interpreting test scores, it's helpful to look at the 25th and 75th percentile scores as well as the median and average. For example, in Group A above, the 25th percentile is 50, while the 75th percentile is 70. This means that half the class scores between those two scores. But in Group B, the 25th and 75th percentiles are 60 and 95, respectively.

Thus, a parent whose kid got a score of 60 would know, if his kid were in Group A, that his kid was indeed scoring near the middle of the curve. But if his kid were in Group A, that would mean his kid was one of the bottom performers in the class, even though a score of 60 is at the 50th percentile in both groups.

I think this next statement by Ms. Dwyer is also confusing:

She said because a list of scores showing that many students do perform in the higher percentile ranges is never reported, it leads those outside education to believe that all children are doing poorly.

Um, there have to be students performing in the higher percentiles, so it's hard for me to understand how the scores in the higher percentiles could not be reported. And whether or not all children are doing poorly has nothing to do with ranking them using percentiles. If in fact every student is doing poorly, a kid with a score of 60 out of 100 could be at the 95th percentile. If it's a bell-shaped curve, that kid is likely to be near the 50th percentile. Every test taker will get assigned a percentile rank, but that only measures how they've done in relation to one another, not in relation to the material on the test.

It's hard to see what she means here, unless she is using "percentiles" incorrectly again and is claiming that scores in the upper categories of scores are not reported. But again, it's hard to believe that that is true, or if true, how it would affect public perception. If the newspaper reports only that 70% of students scored Below Basic on a test, so what? No amount of reporting that 30% scored higher than that will dispel the notion that most of the kids are doing poorly.

But, enough of the statistics lecture. The article winds up by noting that others see testing as a necessary evil, and the most they'll say is that they're not against it:

Merry Naviaux, the academic resources teacher for the Ralston School District and president of the Ralston Education Association, said she sees standardized tests as a necessary tool.

"We need to have some kind of testing," she said. "I'm not against it. I think it is really important because it gives you baseline information that is critical to making decisions."

If opinions on testing really vary a lot, and I think they do, couldn't the reporter have found one teacher or parent to make an unequivocally positive comment on testing?

Junior FCATs soon to come?

In Palm Beach County, FL, first- and second-graders might be gearing up to take junior versions of the FCAT:

School district officials are reviewing whether to require first- and second-graders to take a standardized exam that measures some of the same reading and math skills as the Florida Comprehensive Assessment Test.

There's a lot at stake when students take the FCAT, starting in the third grade. If they fail the reading portion, they can be held back. Their test scores also help determine a school's grade and whether teachers get bonuses.

Proponents see early testing as a way to prepare students for the FCAT before the major consequences kick in. But the idea concerns some educators and parents, who say students in the youngest grades shouldn't be subjected to the pressure of standardized testing.

I see both sides of the story here. Certainly, it's not a good sign if there's really no pressure on teachers to teach reading skills, nor accountability for those skills, until they're going to be tested in the third grade. On the other hand, for a long time the conventional wisdom in psychometrics is that anyone younger than 9 or so shouldn't be tested with conventional standardized tests.

I also don't know of a great deal of validity or reliability research on children that young, although standardized tests like the Stanford Achievement Test do exist for first- and second-graders. From what I can tell from their website, the content is appropriate for kids that age, and there apparently are no strict time limits.

The district already requires a test called Reading Running Records for first- and second-graders. In this test, a student reads a book aloud to a teacher and then summarizes it. The grading is generally seen as subjective and doesn't prepare a student for the FCAT, district officials said.

Broward County has been giving the Stanford Achievement Test to first- and second-graders for years, said Anne Dilgen, Broward's director of student assessment. She said the district has found it a useful tool to predict how well students will do on the FCAT. But it's not an overly stressful test, she said. Students don't have to fill out bubbles, and they don't have strict time limits.

The usual stress-related incidents are what cause some to oppose the testing idea:

Ann Faraone, principal of Calusa Elementary in Boca Raton, used to be an elementary school principal in New York, where testing was given to first- and second-graders. She would rather not see that done here.

She said she has seen students who were so nervous about standardized tests that they got sick.

Yes, but some kids get nervous about recess, or math, or school in general, and the teachers try to help them through it. I don't want to sound uncaring, but the tests given to the first- and second-graders would not be as difficult or high-stress as the test for older kids, so I don't support the assumption that because a few kids get stressed, the tests shouldn't be used for any of them.

What's more, introducing them to testing as early as possible, and in as easy a way as possible, might help them later on.

Virginia educators weigh in on the SOLs

In general, teachers and school administrators in Virginia believe the Standards of Learning assessment have helped them to identify where students need extra help. There are some concerns, though:

The biggest negative cited by teachers was the opportunities for student enrichment. Nearly half of those surveyed, 48 percent, said the SOLs weaken their abilities to provide enrichment for students who have clearly demonstrated that they are performing above the minimum benchmarks. Several teachers added that they felt the SOLs stifle their creativity.

The state superintendent made an interesting reply to these concerns:

Jo Lynne DeMary, the state superintendent of public instruction, said the report confirmed much of what she has heard in her travels to schools across Virginia. She said she would like to investigate further what the teachers mean.

"Is it the content that is restraining you? Is it your lack of confidence to teach the material?" she said. "Through our professional development, I think we can look at more creative strategies for teaching the content embedded in the SOLs."

Interesting to see Ms. DeMary isn't buying the line that standardized tests are, by definition, the enemies of creative teaching. Wonder how VA's teachers will respond to that "lack of confidence" line?

I find this part particularly gratifying:

Furthermore, principals in schools with the most challenging demographics - that is, with the highest concentrations of poverty, underqualified teachers and adults who are not college-educated - indicate that they have received the most benefits from the SOLs.

One principal, for instance, told JLARC staff that the SOLs have given a structure to education that has "made schools determine what is important and what is fluff."

Of course, those who oppose the SOLs insist that "increased structure" is synonymous with "teaching to the test" and "no more fun in the classroom."

And hey! Here's my buddy, Professor Cizek, with his viewpoint:

Gregory J. Cizek, a professor of educational measurement and evaluation at the University of North Carolina, agrees to a point [that the tests are useful for diagnostic purposes]. He said that an individual student's weaknesses in given subjects, such as math or English, can be determined by the SOLs. But a student cannot be judged as weak or strong in a given area, such as fractions, by a few questions.

"If what they are doing is using the results to tailor their instruction in classes at the building and grade levels, that's really good," Cizek said. "If they are looking for good diagnostic information on an individual student in a specific area, these tests are not designed to do it."

The "national" speaker gets in over his head

Oh geez, our old favorite anti-testing crackpot Alfie Kohn is in the news again because of a speech he gave at Roger Williams University. Let's adjust our tinfoil hats and tune in to his frequency:

Alfie Kohn is swimming against the tide, and he would like to encourage others to take the plunge. The national speaker and author, who has twice spoken on the Oprah Winfrey show, has taken a strong stand against mandated standardized testing...

OOoohh, Oprah. I'd say that credential automatically gives his argument more merit than the conflicting arguments of, say, teachers on the front lines in schools, or parents who want public schools to improve, or test developers who work hard to create valid assessments, wouldn't you? I mean, here I am with this Ph.D. and this blog, and I've never been on Oprah.

He maintains that children learn at different rates, and to mandate a certain level of performance with students at certain ages sounds good in theory, but actually puts more stress on teachers and students and results in less, rather than more, actual learning.

As opposed to the pre-accountability system of letting children "learn at their own pace," which has resulted in plummeting SAT scores and increasing numbers of remedial classes in college.

The event was part of the newly launched Institute for Teaching and Learning at RWU, a joint initiative between the university's school of education and Bristol Warren Regional Schools. Mr. Kohn's audience included more than a dozen teachers and administrators from the district. While many of these agreed, at least in part, with his statements, there was a reluctance to go on record due to the political nature of the event.

Wow, more than a dozen? Like say, thirteen or so? And even those who showed up wanted to remain anonymous? Doesn't say much for Mr. Kohn's ability to sway a crowd, does it?

To illustrate his point, Mr. Kohn named a study in which two groups of educators were asked to teach fourth grade students. One set of teachers was told it would be held accountable for raising standards in the classroom. The other set was instructed only to facilitate students' understanding. The first group did far worse. He asked the educators to brainstorm why this might be so, and the answers varied from pressure on the teachers to a larger focus on teaching than learning.

And which study was this? Did the reporter bother to find out? Did Mr. Kohn bother to name it? Who organized this study? Where was it published? And was the one group of teachers who were instructed to "facilitate understanding" given training in doing so, and the other group given no instruction as to better help children learn the basic skills? And even if this was a valid study, does it prove that holding teachers accountable for performance is mutually exclusive from training teachers to help children understand concepts?

No, it does not. If valid, the study says what anyone with common sense knows, which is that you can't implement testing and accountability without also implementing changes in the educational process. Mr. Kohn's view of testing as incompatible with eduation unfortunately colors his entire line of reasoning.

(P.S. - I can't resist adding here that two of my readers wondered how, in this study, the difference in performance was measured between students of teachers told to "facilitate understanding" and students of teachers told they would be held accountable. Was that difference measured by - gasp! - a standardized test? It most likely was. Good catch, guys.)

Mr. Kohn said it's just not possible for a yearly test, which, of necessity, measures facts rather than more general learning, to give an accurate measure of the quality of the teaching. Because the standardized tests are designed so that the majority of students will not excel, "hard" is equated with "good," he said.

No offense to my more sensitive readers, but this is so much horse puckey. There's no evidence whatsoever that standardized tests are designed so that the majority of the students will not do well, so naturally Mr. Kohn provides no data to back up his claim. The fact that many students do not do well on these tests are a harsh indication that, in many schools, "general learning" is valued over facts, although without facts, what are those who are learning in general supposed to be learning?

So questions are often asked that would more likely be learned outside the classroom, on educational vacations or thoughtful conversations around the dinner table. This result is schools with a larger minority population and those with lower incomes routinely score lower on the tests.

If anyone knows what these two sentences mean, or why they're supposed to logically relate to one another, email me and clue me in. I'm mystified.

"If you tell me how many kids in your school are on free and reduced lunch and answer a couple of other questions about socio-economic status, I will predict with chilling accuracy what your school's test results will be," he said.

Any accuracy in that prediction exists because students who are poor often live in underfunded districts with failing schools that are infested with the worst kinds of teachers and administrators - those who believe minority children should not be challenged because they cannot achieve, those who believe that it's okay for eighth-graders to not understand basic grammar, those who believe that basic math skills are unnecessary as long as calculators exist. The only "chilling" part here is that too many people have for too long accepted this scenario as normal. The stark standardized test results no longer allow them to do so.

If educators and parents do not protest, the situation will worsen rather than improve, Mr. Kohn said. In the short term, he suggests prepping for the test as creatively as possible, then getting back to the real teaching. And in the long term, there should be an all-out fight for change, i.e., sending letters to the editor of newspapers, speaking to the school board and contacting state and U.S. legislators.

Change for what? If teachers prep for tests creatively and then teach in a way that allows their young charges to do well on these tests of basic skills, what's the problem?

Unless I'm going blind, there's absolutely no substance here that shows testing is incompatible with good teaching. Kohn is all hat and no cattle. Why the man keeps garnering news coverage, I'll never know.

Another day, another testing protest

Parents of kids at a Montessori magnet school are threatening to boycott the FCAT:

Parental dissatisfaction with the FCAT and the state's school grading formula isn't new. Critics have said that the test unfairly penalizes minority students, who are more likely to be retained or prevented from graduating because they haven't met minimum testing standards.

I suppose these critics have never considered the argument that certain schools are doing the penalizing, by failing to educate these minority youth, either through low standards or poor teaching skills.

Others complain that school is more boring when instructors must ''teach to the test.'' And they say that the state's A-to-F letter grades unfairly penalize low-income schools.

The test gets blamed when teachers are boring in class? That's a new one on me.

But, officials say, no one in Florida has ever organized a sit-out.

If fewer than 95 percent of Virginia Shuman Young students take the test, the school would not be eligible for an A grade from the state. That means 20 to 25 students could sabotage the Fort Lauderdale magnet school, which has earned an ''A'' for the past four years and has a strong chance of succeeding again.

If the school already has shown it could earn a solid "A", and holding kids out could cause the grade to drop, why on earth do these parents want to boycott?

Said parent Michelle Buckman, who is considering keeping her two sons home during testing: ``It would be a great way to get attention. There is just a general concern that we're going in the wrong direction with curriculum. We believe the testing is encroaching on our Montessori curriculum.''

The Montessori theory is that students learn best through asking questions and exploring rather than listening and accepting right and wrong answers.

Can't they "get attention" in a way that won't cause the school to lose money? And while I know little about the Montessori style of education, I have never quite understood the argument that the educational concepts of (1) children learning through asking questions and (2) children learning by listening are somehow mutually exclusive, or the argument that asking questions/doing is ALWAYS better than listening. The FCAT tests very basic skills, but at least on the multiple-choice items, it doesn't matter how a child arrives at the right answer, be it Montessori-magnet or public-school style.

The school's already doing well, so why sabotage it with a vendetta against the FCAT?

Putting the testing squeeze on vouchers

Florida legislators are debating whether to test students who attend schools through voucher programs:

How well are students who get publicly financed school vouchers doing? For four years, the answer has been: It's none of the public's business.

This could start to change, however, as early as next week, when the Senate Education Committee is scheduled to take its first look at a voucher "reform" bill that will likely include a testing provision.

However, the article predicts that, instead of the FCAT, voucher-taking schools may use a different, national standardized test. The hope is that an "independent" party will report the results, which will not be disaggregated by school. This defeats the purpose of seeing whether individual voucher-accepting schools are doing well, unlike Florida's public schools, which are compared to one another every year.

Why the double standard? Become some voucher supporters dislike testing, and some argue that the only "accountability" that is required is assessment of parental satisfaction. But isn't part of being a satisfied parent knowing that your kid is doing well in school? And isn't testing part of that?

Lobbing snowballs instead of filling in bubbles

You know, sometimes the anti-testing reflex is so strong that people don't even catch themselves making a really silly comment. I mean, what was the editor of this ABC News Report thinking?

Many schoolchildren got an early start to their weekend: Dozens of schools closed early on Friday. Some high school students got a lucky break when several schools canceled Saturday's scheduled Scholastic Aptitude Tests.

Emphasis mine. As Best of the Web puts it:

Yeah, we suppose that's a "lucky break"--unless they want to go to college!

USA Today, on the other hand, recognized the SAT snow issue for the hassle that it is:

Foul weather canceled long-awaited college entrance exams on Saturday for thousands of high school students up and down the East Coast, forcing them to reschedule their Scholastic Aptitude Tests (SATs) until either five days before Christmas or sometime in January.

Most students taking the test in December are high school seniors in the midst of applying to colleges. For many, the Dec. 6 test represented one of their last opportunities to record a good score on college applications.

Trust me, ABC, those snowbound kids don't consider themselves "lucky."

Getting a leg up with Spanish skills at UC

The LA Times reports that fluent Spanish speakers once had a leg up in admissions at the Univeristy of California, because the SAT II Subject Tests were weighted twice as much as the SAT I in admissions, and Spanish was one of the permissible entrance exams. Thus, the ability to speak Spanish could help compensate a great deal for a poor performance on one of the other exams.

However, that advantange for native Spanish-speakers is getting ready to disappear:

In the UC eligibility formula — a weighted index combining test scores and grade-point averages — a high score on one of three subject tests, known as SAT IIs, can be a saving grace for an otherwise weak applicant...Designed for nonnative speakers, Spanish is among the offerings [for the SAT II]. , the tests are available to everyone but are typically easy points for students...[who have Spanish as a first language]. But the advantage to native Spanish speakers is about to be radically reduced.

Changes approved by the UC regents this summer and set to take effect in 2006 give far less weight to SAT IIs. The changes are likely to have the greatest effect on applicants who score poorly on the SAT I, a broad test of verbal and math skills.

It's not surprising that the SAT reweighting will hurt some, especially those who depended on the SAT II Spanish subject exam for help. What is somewhat surprising is the decision to reweight. For starters, some or all of the SAT II exams may be, for some schools, a better predictor of first-year college grades than the SAT I.

On the other hand, ability to speak Spanish is not currently a required skill for college class performance in the US, and the SAT II Spanish Subject Test was, as I emphasized above, designed for non-native speakers, which suggests that it is ridiculously easy for native speakers. With the existing weighting, fluent Spanish is worth as much in admissions as evidence great English and mathematical skills, whereas it is highly unlikely that skill in Spanish is as good a predictor of college performance as English and math. If it's predictive at all, I'm sure it predicts different for native vs. non-native speakers.

This last assumption, if true, merits removing this particular exam as an option for native speakers. So why reweight all the exams? Well, before we can find that out, we have to wade through a series of dire statements about the negative impact of reweighting:

According to a Times analysis of UC data, Latinos who score 1,000 points or fewer on the SAT I will likely go from having about the same chance for admission as low-scoring whites and Asians to being at a disadvantage...

Even now, Latinos are significantly underrepresented on UC campuses. They account for 42% of 18- and 19-year-olds statewide, but only about 16% of UC admissions last year...

Okay, but I assume the Board of Regents knew, or could guess, all of this. So why change the admissions system?

Blacks and Latinos, by and large, score well below Asian Americans and whites on SAT I and other standardized tests. The theories for why that occurs range from test bias to a range of socioeconomic, cultural and historic factors.

I'd insert here the comment that "Knowledgeable or sensible theories for why that occurs range across socioeconomic, cultural, and historic factors," but y'all knew I was going to say that anyway.

What is clear is that for the last three years, the Spanish test has allowed Latinos who score poorly on the SAT I but are fluent in Spanish to compete with low-scoring whites and Asians, while blacks are still left behind...

Ah, now we're getting somewhere. The current weighting helps only some minorities. The article points out that black applicants often have had the same socioeconomic hardships to overcome as the Latino students, yet only the Latino students receive the double-boost of the Spanish exam.

A range of SAT II language tests is offered, from Chinese to Hebrew. They tend to attract native speakers, who often push up the average scores well beyond those for such tests as science or history. But no test has the same effect as the Spanish exam, by far the most popular test chosen by Californians.

Among UC applicants last year who scored 1000 or less on the SAT I, Latinos averaged 629 out of 800 on their third SAT II, according to university data analyzed by the Times. Asians averaged 488, whites 461 and blacks 438.

And then those SAT II scores are doubled, so you have Latinos with a score of 1258 going into the admissions index, as opposed to 976, 922, and 876 for the other three groups, respectively. The result is that Latino students with low SAT I scores have been, since 2001, admitted at vastly different rates than other groups. For those scoring below 1000 on the SAT (which is, give or take, the average score across the nation), 65% of Latinos were admitted vs. 49% of blacks.

The article mentions research from the late 1990's suggesting that "subject tests in general are a better predictor of college performance than the SAT I." I assume the research mentioned is this paper, which I've seen before. While this paper does show that the SAT II is a better predictor than the SAT I for UC's students, it does not break down the SAT II by exam subject, and nowhere does it suggest that the SAT II score should be weighted twice what the SAT I is weighted.

The policy went into effect in 2001 before anyone realized the extent to which the SAT II Spanish Subject Exam would influence admissions decisions. And now with the new, improved, upcoming SAT I (which includes an essay), the SAT II test impact will diminish.

"If there is a person out there who scores poorly on every single test and extraordinarily well on one test … I'm not sure that is the best student for the university," Widaman said.

I'm sure there are plenty of folks who would disagree with Widaman, despite the fact that (a) he has a point, and (b) giving native Spanish-speakers an edge in admissions into an English-speaking university system doesn't seem fair at all:

At Locke High School, 62% of the students are Latino, nearly all the rest are black, and standardized test scores are among the worst in the state — except on the Spanish exam. Changing the eligibility formula could increase the difficulty of sending such students to UC campuses.

"Our kids are horrible test takers," said John Mandell, college counselor at Locke. But on these exams, "they do well." He encourages native Spanish speakers to seize the advantage by taking one of the two Spanish exams offered.

Why? Their ability to speak Spanish well may not indicate how capable they are of completing English course work. Why not focus the energy on figuring out why the students are "horrible test takers" and correct that? The fact that they all do well on the SAT II Spanish suggests that they are familiar with the multiple-choice format and are not cowed by the idea of taking standardized tests.

My hunch is that these students aren't learning English very effectively at Locke; thus, they do poorly on exams which requires them to translate everything into English before the items can be tackled. But in the long run, shouldn't making sure the students learn English be more important to Locke than getting their graduates into an environment for which they might be ill-prepared?

From his windowless office, Mandell ticked off recent SAT II Spanish scores among Locke students: three perfect 800s and several more scores above 700.

Whereas only one student broke the 1000 record on the SAT I. These test score discrepancies make it painfully clear that the Spanish these kids learn at home seem to be just about all they've learned during their high school career.

An analogy is provided in support of the double-weighting: "disallowing Spanish speakers from showing off prowess in their native tongue would be akin to barring the children of physicists from taking the physics test."

Not a good analogy. If the point is whether or not those children can tackle college-level physics classes, then admit the highest scorers, regardless of background. But the point of college admissions in the US is whether or not the admittees can handle college-level classes taught in English, so why should we assume that students are ready for UC merely because of the fact that they are fluent in Spanish?

I mean, let's not forget the predictive validity issue here; the anecdotes which suggest a couple of capable students needed a leg up are just that - anecdotes. I want to know how many of those admitted with below-1000 SAT I scores and sky-high SAT II Spanish scores are still in the UC system, and doing well - and not enrolled in remedial classwork.

Joanne Jacobs comments as well.

New tests for New York City

New "measuring sticks" for students in the Big Apple:

Schools Chancellor Joel I. Klein said yesterday that his office was developing new ways to measure performance in the school system that would offer a far more sophisticated analysis than is currently gleaned from standardized test results.

Mr. Klein said the new accountability system would be aimed at judging every aspect of the school system, from the performance of teachers and students to that of administrators.

My guess is some teachers and administrators have nightmares about this very same circumstance...

On the instructional side, much of the chancellor's plans are focusing on a relatively new way of judging the performance of students and teachers, called value-added assessment...

The system can also be used to determine the effectiveness of teachers by showing whose students consistently achieve at a faster or slower pace, regardless of where each student starts out.

The teachers' union president claims that the concept has "real promise" but will fail because Klein didn't seek the advice of teachers earlier. Apparently, this concept is far too "difficult" and "complicated" to implement, despite the fact that it has already been implemented in a dozen states so far.

Mr. Klein said that the current system of accountability often failed to recognize excellent work by teachers in the toughest schools. "You could be working in the most challenging school doing incredible work," he said. But "you aren't getting the kind of recognition that your colleagues are."...

Mr. Klein said that another measure of accountability might be customer satisfaction with the schools based on surveys of parents. "It's not a single metric," he said. "It's a system of accountability for the mayor, for me, for everybody."

Read more about value-added assessment here and here.

Flurries expected in Connecticut

We might get our first winter storm up here in the Northeast this weekend. In Connecticut, they're expecting flurries of snow - and flurries of school-accountability data:

From report cards to warning lists, the state will release a flurry of data this month to parents and educators describing how Connecticut schools are measuring up against the federal No Child Left Behind law.

The state got a taste of the law's reach this summer, when 149 - or one in five - elementary and middle schools in the state were stamped for not making adequate yearly progress. Thomas Murphy, spokesman for the state Department of Education, said officials expect the percentage of high schools identified to be higher.

I particularly like the one quote from a school superintendent, below:

For parents trying to wade their way through the volumes of numbers, regulations surrounding school choice and definitions of No Child Left Behind buzzwords like "adequate yearly progress," schools are trying to help pound out the specifics.

"Parents understand only what they see on a Web page, see in a newspaper, hear on television. I get maybe one request a month to transfer to another school system, or have their child tutored at home based on what they perceive to be the law," he said.

Emphasis mine. Hey, on some web pages, parents can get useful information, you know? It's great that the schools are holding workshops, but why not have better websites or even weblogs to help explain this overload of data to parents?

More fretting about Head Start exams

The worries over the Head Start testing program just keep a-comin'. The previous high anxiety article related to this program featured a tyke named Nate; this time, our object of concern is Joe:

In a small room in east Portland, the world of standardized tests -- and of educational accountability -- has reached down and tapped Joe on the shoulder. He's 4 years old. And as he sits across the table from the woman with the pencil and the score sheet -- counting the blocks on a page, trying to name letters -- he unwittingly finds himself at the center of a national debate.

Education leaders in the Bush administration believe that many Head Start programs -- part of the 38-year-old national program that tries to prepare preschool children from poor families for kindergarten -- don't do their jobs well enough. So, this fall, they are mandating standardized tests of the 450,000 4-year-olds in the nation's more than 2,500 Head Start programs...

So far, so good. A neutral enough introduction to the issue, which is indeed thorny. If Head Start is indeed meant to give kids a head start on academic issues, it's not unreasonable to develop some way to assess the quality of the academic instruction. Problem is, some Head Start supporters believe the tests are actually at odds with the heart of the program:

...many Head Start officials, including some at the three Head Start programs in Portland, say the tests are useless, at best, and could end up threatening the good work that Head Start programs do.

Samuel Meisels, a Chicago specialist in early childhood education, said the 20-minute standardized test -- formulated in a matter of months this year -- "is incredibly narrow ... a very, very limited sample of children's knowledge of vocabulary, letters and math."

I'm not surprised the test is narrow; is Meisel's suggesting that we make the test longer? I don't think so. I think the issue here is that the Head Start programs will be judged in part based on a narrow measurement of ability (that is questionable to begin with based on the ages of the test takers), and that's a valid concern.

However, if there's a different and better way to test these kids, I see no reason not to do so. I also was unaware that the Head Start programs would be judged entirely based on test results. In fact, I doubt that's the case. The test has been mandated to help improve Head Start program quality, not just to grade and punish different programs. One Head Start advocate is quoted as saying that this testing requirement (which parents can opt out of, by the way) is just a means to destroy the program altogether, which doesn't jibe with the following:

Federal Head Start officials say it's ridiculous to think that they want to end Head Start. "The idea that you would somehow dismantle, (or) signal that you want to dismantle the program, when you have proposed a $148 million increase in its budget seems contradictory," said Windy Hill, associate commissioner of the federal Head Start Bureau.

"In terms of this being some grand conspiracy ... there is not a conspiracy," Hill said. "There is a commitment to make sure that the $6.7 billion that the taxpayers provide" is being spent effectively.

Six point seven billion is a lot of money, and that money came from all of us who pay taxes. So who says we can't start finding some way to see if that money is being used to good purpose, instead of being mismanaged or thrown away outright? If these tests don't give us good data, let's try something else. But it's absurd to say that the desire to see if a system is functioning efficiently is the same as the desire to destroy that system.

Overall, though this article is much more balanced and neutral than the NYTimes article that I blogged a while back. Not a bad job.

Oh, and what about Joe?

Which brings things back to that small room, and Joe, and the woman with the pencil asking him questions. Joe (not his real name -- his parents asked that his name not be used) sits in a tiny chair across a small table from Head Start test proctor Tina Williams, his short legs swinging free, not quite reaching the floor. He's able to name many of the letters of the alphabet that Williams shows him. For another question, he counts 13 blocks on a page before his counting gets confused.

....Joe's place at the center of the debate has ended -- at least until next spring. As he finishes the test, Williams tells him he did a great job and lets him choose an animal sticker as a reward. Joe ponders, chooses a small alligator sticker, smushes it proudly on his bright red T-shirt and happily walks out of the room.

Releasing FCAT items to parents as "quality control"?

A St. Petersburg Times Editorial assumes that Florida's Governor Bush is "gloating" over the recent decision to keep test items out of the hands of parents:

An appeals court has now determined that Florida parents don't have a right to see where their students are going wrong on standardized tests, but Gov. Jeb Bush shouldn't be so eager to gloat. His win comes at the expense of students who are being held back without really knowing why.

Why assume Governor Bush is "gloating" about this? Withholding test items is nothing to gloat about; it's merely standard testing practices with companies and/or school districts that do not have the money to create new forms each year.

The governor portrayed his opposition to disclosing FCAT test materials as consistent with "the Department of Education's 20-year policy on test confidentiality," but that's a little disingenuous. Until four years ago, DOE never used a standardized test to grade and punish schools. Until last spring, it never used a state test of such complexity to decide whether high school seniors could graduate. Until this fall, it never used a state test to decide whether third-graders should be promoted or retained.

The switch to using a test for these methods does not necessarily compel the test developers to release test forms. High stakes do not require disclosure. Some companies do disclose test items, but only because they have the staffing and the funding to create new test forms each year that have been equated and checked for bias and validity issues. I've commented on the lack of understanding of the financial issues surrounding testing before.

And a test "of such complexity" for seniors? Please. Here are 10th-grade Reading sample items. The reading passages are appropriately difficult, but the questions are often very easy. For the first passage, simply knowing the meanings of the words "surly" and "maxim" gets a student two right answers out of eight multiple-choice items. The specifications for these passages indicate that "how-to" articles and advertisements are appropriate "forms of informational text" on the exam; this suggests that some reading passages might be very, very easy.

The technical report (p. 16) notes that the p-values for the 2000 Operational FCAT are distributed as might be expected for a test of basic skills. For example, on the 10th-grade reading test, there are a few hard items, but 75% of the items were answered correctly by at least 59% of the examinees.

And did I mention that Florida's students get six chances in high school to pass the 10th-grade FCAT?

The issue of using a test to promote third-graders is thornier. The little kids get fewer passes at the test, and one could legitimately argue that students that young are not disciplined enough to deal with a high-stakes standardized test, nor has anyone come up with a solution for what to do with the ones who flunk this test repeatedly.

Because Florida has so dramatically increased the stakes associated with one test, it owes students and teachers a better understanding of how they are performing. It also owes them better assurance that testing error didn't lead to grave consequences in their lives.

I agree. But this "better assurance" doesn't mean driving the costs of tests up astronomically by releasing items to parents. It means implementing quality control mechanisms to ensure that the test is error-free before it is administered, to ensure that the scoring and score reporting processes are bug-free, and to ensure that scores are being interpreted correctly (i.e., with a standard error of measurement band that reflects the reliability of the test).

Comparing schools in Nebraska

Nebraska's state report card uses "Adequate Yearly Progress", or AYP, to judge how well schools are doing. However, only in Nebraska are schools allowed to create their own assessments for this type of monitoring; only the writing portion of the tests are the same across schools. Does this means it's impossible to actually compare Nebraskan schools to one another?

When Nebraska's State Report Card is released on Monday, most of the attention will likely focus on which schools made "adequate yearly progress" and which did not....Superintendent Steve Joel has been warning for weeks that the news for the Grand Island school district will not be all good when AYP results are released on Monday...

However, he noted that most people will inevitably compare results...He also said that Nebraska Department of Education officials believe that the results between schools are at least roughly comparable because each school district is supposed to use the same set of criteria when developing their assessments.

If the purpose is to compare schools, though, why allow each school to develop its own assessment? Schools that have not developed assessments, in fact, use the ITBS (Iowa Test of Basic Skills), and this may be part of why some are not yet making AYP; the ITBS may be more difficult than Nebraska's state standards require.

I also think it's naive to assume that anyone can create a good test as long as the standards are clearly defined. Having good standards is necessary but not sufficient to have a test which measure those (and only those) standards in a reliable and valid method.

Thanksgiving Roundup

Howdy, folks. I'm going to get out of here early today, so I'm going to post just a few entries for you to chew on, then I'll be back sometime on Monday or Tuesday.

----------

Wendy McElroy's got the roundup on the appalling prevalence of zero-tolerance policies in school, which she believes is related to society's willingness to charge ever-younger kids as adults for crimes such as murder and sexual molestation.

----------

Stuart Buck, of The Buck Stops Here, takes issue with a chirpy, optimistic statement by a kindergarten teacher at his son's school. The teacher claimed that "today a schoolchild learns more between the freshman and senior years of high school than our grandparents learned in their entire lives." Stuart's response?

That can't possibly be true. For one thing, there is no meaningful way to measure the total sum of the knowledge that our grandparents learned in their entire lives. And just think about it: Do you really think that our grandparents learned less about the world in 70 or 80 years than today's high-schooler does in 4? Have you met any current high-schoolers? Do they really seem more knowledgable than their grandparents about anything beyond computers and cell phones and Eminem?

----------

The NEA is apparently goading Oregon Governor Ted Kulongoski to sue the Bush administration over the "hoax" that is the No Child Left Behind Act.

The National Education Association has been looking since July for a state to sue the Bush administration, arguing that the law requires sweeping changes in schools without paying for them. No state has signed on, despite widespread complaints by educators that the law requires too much of schools.

But some fear that suing the Bush administration could backfire on Oregon.

Any Oregonian readers out there got an opinion about this?

----------

Sarah Lawrence College will no longer require the SAT. Hey, if it's not right for your school - and your school is willing to spend lots of time on each application - then don't use it. Larger universities, though, will almost certainly continue to retain the test as a way of winnowing down the massive number of applications they receive each year.

----------

Joan Ryan, in her argument for smaller schools, makes an insightful comment about the balance that is needed between teaching and testing:

I understand and even support the rationale for spending money on standardized testing: We have to measure students' knowledge so we can know which schools are working and which aren't. But with limited resources, the priority ought to be creating small schools and training teachers. If so many of our children are starving academically, doesn't it make more sense to put our money first into feeding them, then into weighing them?

Testing is not the be-all end-all of education, nor should it be. The problem is, it isn't until test scores are produced that some schools can be convinced they need to change.

----------

Jay Mathews of the Washington Post summarizes a list of "wild ideas" for simplifying the college admissions process. Cliff Sojgren is the author of the ideas, which include requiring high schools to provide enough information so that grades can be compared for students from different schools, eliminating the entrance essay, and only using SAT/ACT scores when they are average or above.

I have to disagree with Sojgren's idea that low SATs should be ignored. Certainly other factors can come into play, but what if, for example, a group of students with high grades but low SAT scores all come from one school? That information could be part of the factors used to judge how inflated the grades are from that school. And Sojgren's plan to rate schools will come under just as much fire from those who cry racism/classism as the SATs do now. If the A's given by teachers with advanced degrees are "worth more" in this new admissions process, you know that any school with a high percentage of teachers without advanced degrees is going to cry racism if those teachers are minorities, or classism if those teachers live in a poor neighborhood.

But it's food for thought, nevertheless. And speaking of food, well, I've got to finish up work so that I can drive 11 hours tomorrow to get some really good food.

Ah, Southern Thanksgivings. I hope yours is as blessed and stuffed with love and calories as mine will be.

Thank God for California

It used to be, while I was living in South or North Carolina, that any discussion of the state's test scores was ended with, "Thank God for Mississippi." The southern states are known for being bottom-feeders when it comes to overall test scores, especially when the performance of poor children is examined.

Well, now SC and NC can say, "Thank God for California":

It has often been comforting for education watchers to ascribe such gaps to California's high level of poverty among minority students. But the NAEP data don't support that old saw. Other states have poor children in large numbers, and if NAEP is an indicator, they do much better by them than we do.

It's a sad day when Californians can look at test scores and say, "thank God for Alabama and Mississippi."

(Found via Education Weak.)

Trials and Tribulations on the TAKS

WOAI in Texas reports that more than half of San Antonio's 11th-graders flunked the new TAKS, which replaces the TAAS. They won't be disadvantaged this year, but next year high school students will have to pass this exam to graduate.

WOAI responded by hiring four adults - "a city councilman, a former judge, a DJ and a school board president" - to take the TAKS, so we could all see how well they did. The judge and the school board president had actually taken the TAAS before and declared the TAKS to be noticeably more difficult:

Jamie of Mix 96.1 says she was on the honor roll, in AP classes throughout school and in the National Honors Society. She was a lot of talk before the test, but when she turned down the volume she sounded more like this: "I forgot what these little numbers meant."

This is the second sitting for NISD board president Bobby Blount and former judge Cyndi Krier. They took the TAAS test for news 4 WOAI four years ago, and passed with flying colors.

The results? The DJ flunked the math portion; the councilman flunked both math and English. The school board member passed both sections, while the judge ended up flunking math while making a near-perfect score on English.

How did the DJ interpret her flunking math score?

Jamie says it is. "Kids did you hear me? You don't need to learn math like me. You can still be successful and do bad on math."

WOAI disagrees:

But what you can't do is get a diploma without passing the test. Educators say they saw the same kind of failure rates and complaints when they introduced the TAAS test. By the time it was retired, those teachers say, the TAAS test was considered too easy. The idea is that bigger challenges create brighter students.

Here are the statewide 11th-grade results, in case you're interested.

If I read the table heading correctly, what's listed are students grouped into (1) those who met a standard that was set at two standard errors of measurement (SEMs) below the panel's recommendation, (2) those who met the standard that was 1 SEM lower, (3) those who met the panel recommendation, and (4) those with a "Commended Performance" that is presumably somewhere above the panel recommendation.

Take a look at the first line, for all 11th-grade students on the math portion (ignoring the spring field test results). The numbers are 68%, 55%, 44%, and 6%. Working backwards, this means that:

Only 6% scored in the "Commended" category, whatever that is.

A total of 44% scored at the panel recommendation or above, which means that, had the cutpoint been set there, 53% would have failed.

A total of 68% passed at a cutpoint set two SEMs below the panel marker, which means that a whopping 32% of all examinees are more than two standard errors of measurement below the cutpoint. Those students aren't just failing - they're failing miserably, because they're not within the 95% error range (based on the reliability of the test). Thus, that 32% is far enough away from the panel recommendation that it's highly unlikely they would pass upon retesting (assuming no change in true ability).

The English scores look even more bimodal; when you go from at panel cutpoint to 2 SEMs below, you only get an increase from 61% to 69% of the students. This means, essentially, that around 60% are passing easily, around 30% are failing miserably, and there's relatively few students - only 10% of examinees - in between.

Update: Here's an article from Education Week that criticizes the TAAS in comparison with Texas' NAEP scores. The writer does not seem optimistic about the usefulness of the TAKS, either.

Does high stakes testing make teachers cheat?

The Harvard Crimson reports on a new study that concludes that "teachers and school officials cheat in administering standardized tests in a minimum of 4 to 5 percent of elementary school classrooms." Although the headline blares that "High stakes tests lead to cheating," the conclusion is probably not what testing opponents hoped for:

The authors, Kennedy School Assistant Professor of Public Policy Brian A. Jacob and University of Chicago Professor Steven D. Levitt, concluded that local policies attributing more weight to standardized testing made it more likely that teachers would cheat...

“In the future, we hope to try to prevent this kind of behavior by having external monitors, as well as by performing random audits, to discourage these kinds of acts from reoccurring.”

In other words, we notice cheating more now because tests carry more weight, and while the raising of stakes might cause some teachers to cheat, that increased prevalence isn't necessarily an indictment of the tests, or the stakes. This merely suggests that stakes were raised, but the corresponding security controls were not; in any such situation, I'd think you'd see more cheating. Dr. Jacob's conclusion is correct; how do we institute security measures to prevent this from happening in the future?

Here are abstracts for Levitt's recent papers on catching cheating teachers and the prevalence of cheating teachers. Some of you may recognize Levitt's name; that's because he's been mentioned here before, as the Chicago professor who's figured out new ways to detect cheating behavior. Apparently the Chicago public schools provide quite a rich dataset for detecting such behavior.

Test scores not adding up in Georgia

Online Athens reports that school officials in Clarke County (GA) are concerned about sagging - and conflicting - math scores among students enrolled in middle and high school:

The Clarke County school board got a snapshot look at district math achievement during its regular monthly meeting Thursday - a chart of standardized test results that showed only 47 percent of eighth-graders met or exceeded the benchmark math score on the state's Criterion-Referenced Competency Test given in the spring. A year earlier, 57 percent had met or exceeded the benchmark as seventh-graders, and 58 percent had done so as sixth-graders in 2001.

In a similar drop, 56 percent of sixth-graders met or exceeded the achievement benchmark on the spring CRCT. Last year, 62 percent of the same class met or exceeded the benchmark as fifth-graders...

Definitely not good news. For some reason, the same cohort that is moving through the county's schools is becoming less likely to meet grade-related benchmarks as they proceed.

Administrators believe the downward trend in test scores through middle school is a symptom of the same lack of math comprehension that leaves many high-school freshmen floundering in algebra class - which they must have to graduate. The struggle for many students has been a concern of some school board members in the past, and Cedar Shoals and Clarke Central high schools have begun offering pre-algebra and Algebra I classes as year-long classes, rather than semester-long, in one attempt to help students.

You mean these high schools were teaching Algebra I within a semester, instead of using an entire year for that? Why the rush, unless what they're calling "pre-algebra" is just part of Algebra I? And move forward three years, and things get more interesting, as passing rates on the high school exit exam rapidly increase:

Math scores skyrocket between eighth-grade CRCTs and the Georgia High School Graduation Test given in the junior year of high school...Superintendent Lewis Holloway told school board members that part of the reason scores jump on the high-school test is because many of the students who have struggled academically have begun dropping out by that point and therefore aren't taking the test. The test also is not very rigorous and is being revamped by the state, school board member Denise Mewborn pointed out.

Well, here are the online content descriptions for every grade for the math CRCTs. Skip to page 69 to peruse the eighth-grade content. The content appears fairly extensive and includes geometric and algebraic concepts. Sample items may be found here; the items seem pretty straightforward.

And here are the content descriptions for the math portion of the Georgia High School Graduation Test. The Geometry and Algebra constructs being measured do not seem to be any more rigorous than those listed for the Grade 8 CRCT.

Here's the student guide from the DOE. Notice that an item which requires a student to calculate the following is listed as having a high cognitive level:

6. "One gallon of paint will cover 800 square feet. How many gallons of paint are needed to cover a wall that is 8 feet high and 200 feet long?"

Why is this considered to be one of the more difficult items on the exam? Because it requires test takers to "know how to find the area of a rectangle and to know when finding the area will help solve a problem."

I think Ms. Mewborn might have a point about the rigors of the exit exam. From my admittedly-cursory examination, it appears that the graduation exam is not much more difficult than the eighth-grade exam, and this could definitely explain the contradictory scores. Students who are failing in math in eighth grade either drop out or have almost three more years to master the eighth-grade material before taking the exit exam.

Coordinated teaching reason for MEAP rise

An "integrated curriculum" and "teaching of thinking skills" are being cited as the reason for the rise in recent MEAP scores:

Michigan Education Assessment Program test results released last month showed gains in many districts in elementary math, social studies and science scores...

"When we studied whales, we had books in the classrooms on whales, and in your social studies you learned about their habitat," Moyer said. "Even your math project might be about whales. If there isn't a connection made, there's no value. If they can see what they are doing in math connects to social studies and reading, there is value."

One school in particular, the Cheney Academy of Math and Science, saw tremendous gains on the MEAP:

Teresa Wilson, principal of Willow Run's Cheney Academy of Math and Science, said that in addition to coordinating curriculum at a particular grade level, teachers at her building align lessons from grade to grade and communicate clearly with each other about students' progress from year to year. Also, the school has a multi-age format, enabling teachers to have the same students for three years in a row.

The percentage of fifth-graders passing the science portion of the MEAP rose from 48 percent in 2002 to 100 percent in 2003...Only 7 percent of fifth-graders at Cheney passed the social studies test in 2002, but 63 percent passed it in 2003.

This suggests the students were starting off with moderate science knowledge and almost no social studies knowledge before entering the Cheney environment, doesn't it? The students at Cheney believe the constant feedback about where they need improvement helps, although the "bagel breaks" they get on test days are certainly popular, too.

More information on the new SAT

Via the Cornell Daily Sun:

The addition of a third 800 point section will be the biggest change to the test -- which will bring the total possible score on the SAT to 2400 points, up from the current 1600. The new writing section will include multiple-choice questions about grammar and word usage as well as an essay. The essay topic will require students to take a position on a given issue and to support their arguments with examples. The essay will occupy about 25 minutes of the test and most likely will be the first task students have to tackle. The multiple-choice questions will test a student's ability to identify errors and make corrections to sentences...

"Writing an essay scares people," he said, adding that most students probably would prefer a multiple-choice format.

If students who consider themselves to be college material are scared of writing essays under a time limit, well, whose fault is that? Here's a related article from the Technician Online that does not support the use of the SAT, but still urges NC schools to develop their own writing requirements in order to improve the quality of the incoming freshman class:

Critics have long charged that the SAT is riddled with problems. Some students do not test well or panic in a timed, standardized test situation, they argue. Others say that test-taking prep programs that are often expensive or inaccessible to all make the test biased against minorities or the lower class.

Instead of placing high emphasis on the SAT writing test, NCSU should consider requiring its own writing requirement as a simple evaluation of writing ability and as an opportunity for students to share part of their lives that is not evident or clear from a simple transcript.

In other words, even if schools do not agree with the timed, standardized writing assessments on the new SAT, it would be foolish to leave out any requirement of writing skill assessment, no matter how "scary" folks find it to be.

Update: And here's yet another article on the difference between the old SAT and the new, which - oddly enough - claims that "some think too much is being made of the SAT changes." GMToday obviously doesn't believe that, or they wouldn't have printed this.

Giving praise to some but not to all

When the goal is to increase test scores, is it wrong to single out those who do well on tests for praise?

When it comes to taking a standardized test of any sort, who wouldn’t balk? For some, it’s the sea of empty ovals bobbing before their eyes that make them bite their fingernails. For others, it’s that they love the math portion of the test but loathe the English part. And for still others, it’s the stress of nailing that final score.

Administrators believed that celebrating those individuals with the highest scores recognized the results of hard work and skills. Students, however, believe the practice needlessly embarrasses those without high scores on the day of their graduation. Students, teachers and administrators plan to meet this month to see if this all should change before the next graduation.

You know, I received a special award on the day of my high school graduation; an award that only four members of my graduating class received. I would have been rather upset if my school officials had chosen the option of not praising the four of us at the graduation ceremony, under the assumption that the remaining 436 members of the class, easily identified as not receiving the award, would have been embarrassed.

I mean, come on. If schools want students to take the exam seriously, then why not let the ones who do well take glory in the results? Removing the test score designations from the graduation booklet gives the clear message that the school is less concerned about praising those who achieve than about protecting those who don't. If that's the case, why should a student care about the test?

A snitty take on NCLB

The Nation, the legendary publication of the left-wing's "snits and quarrels," as P.J. O'Rourke once put it, is acting postively ultra-snitty about standardized testing and the NCLB act. I don't have time to answer every "criticism" (most of which are acts of name-calling and outrageous hyberbole), but I do have to point out a few choice lines, so that you can see what passes for "valid criticism" of education reform from the far left:

It's true that in the past, schools could hide poor performance of, say, special-ed students by averaging it in with that of excellent students. Pulling out the subgroups creates what is called transparency. And that's fine, as far as it goes. But under NCLB, transparency is transmuted into school-bashing. In the words of the North Carolina State Board of Education, "A school's making AYP is an all or nothing prospect. A school will either have 'Yes' or 'No' in this field." One of Palo Alto's top high schools received a scarlet letter because some students skipped the test to study for AP exams.

In other words, transparency is okay, except when it's not. Pulling out subgroups is okay, until schools get slammed on the results. Quite a bit of equivocation here.

And remember, this is all based on how some squirrely kids perform on a standardized test that neither the public nor the educators have a right to examine. In some states a teacher is subject to reprimand or dismissal if she even glances at it. Or tries to comfort a child sobbing over the test.

"Squirrely"? Is that a real word? Is it supposed to be a blanket slam of all US students? What states have standardized exams that no educators ever see; which state withholds all exam results from the public? Can the author here give us a reason that teachers should be allowed to "glance at" live test forms before they are administered? Does he really think we believe teachers have been fired for trying to comfort students who are upset?

States must come up with a plan for achieving 100 percent proficiency by 2013-14, so they set up a grid: Oregon is typical, promising 40 percent proficiency in English/Language Arts in 2002-03, jumping to 60 percent by 2007-08, 80 percent by 2011-12 and 100 percent by 2013-14. Note that they're putting off the utterly fantastic gains until the last years. Maybe they're counting on NCLB's self-destructing by then.

Actually, as Jay Mathews already pointed out, the revision of those targets do not imply the "self-destruction" of the NCLB. What's more, when targets were previously set lower, schools didn't respond. That "100%" is supposed to spur schools on, but NCLB will most likely be modified once schools realize that just aiming to educate 60% or 70% or 80% of their charges is no longer acceptable.

A July press release from the Business Roundtable quotes Joseph Tucci, chairman of the Roundtable's Education and the Workforce Task Force: "You can't manage what you don't measure. No executive can run a business without accurate, granular data that explains what's working and what's not. Our school systems should be no different." Keep those 8-year-old widgets rolling along the conveyor belt! But man does not live by granular data alone. Neither should children, though everywhere music, art and recess are being cut--to make room for more test prep.

Ah, yes, the classic argument that students are dehumanized simply by being measured, and that testing must necessarily replace all other forms of discernment. Never mind that students who can't read at a basic level might not grasp the finer points of art history. Never mind that humans have been measured, evaluated, judged, poked, and prodded for at least the last couple of thousand years, and somehow we still have souls. The valid argument that some of these tests might not be accurate, or reliable, or informative, is here discarded in favor of the idiotic argument that human beings cannot be tested without their humanity being destroyed.

What does that make psychometricians like me, I wonder? God, or the Devil?

When the 99th percentile isn't good enough

Here's a darling little overachiever. Jeffrey Lin, of Plano, TX, earned a perfect score on the ACT but "only" a 1440 on the SAT. This "low" score bothered him, so he took the test again.

Now he has perfect scores on both exams:

Lin retook the [SAT] in October, feeling more relaxed and confident about his answers.

The novelty of his accomplishment is sure to cause a double take when college admissions officers look at his application.

Note: The article lists Lin's perfect ACT score as a 30, which must be a typo; the top score on the ACT is a 36.

The 2003 NAEP scores are out

The 2003 scores for NAEP (National Assessment of Educational Progress, pronounced "nape") have just been released. This exam is often referred to as a "national exam" or "the nation's report card" because it is the only exam taken by students in all 50 states. Scores are not disclosed at the individual student or school level, but scores are reported for the nation, for states, and for specific populations of students. The performance scales are created so that students in different areas can be compared to one another, and test takers can be compared within test sections across years. An overview of the exam may be found here.

Here's the official statement by the group that produces and scores NAEP, the National Assessment Governing Board (NAGB, pronounced "nag-bee"). Thanks to the 2003 results, NAGB is now optimistic about math performance; less so about reading:

Since the year 2000, the last time the NAEP mathematics assessment was given, the students at the bottom have made the greatest improvement. The largest gains have been achieved by fourth grade students in the lowest 10 percent or the lowest quarter of the test score distribution. The lower-scoring students in the 8th grade also have made substantial improvements.

The overall picture is encouraging because not only did the lower-scoring groups improve, but higher-scoring students made gains too, although at a somewhat slower rate.

Test takers in various ethnic groups are the "subgroup populations" to which I referred, above, and these results are indeed encouraging.

In reading, unfortunately, the situation is less clear. This year, 2003, is the first time that a subject has been tested by NAEP two years in a row. It is unrealistic to expect dramatic changes in one year—particularly for the large groups of students in a state or in the nation on which NAEP reports. And the 2003 reading assessment shows very little change from 2002.

The situation recently in 8th grade reading is less positive. Even though there was some gain in 8th grade reading achievement from 1992 to 1998, the overall performance has been essentially flat over the past five years.

As Joanne Jacobs notes, a student's math performance may be more affected by the school's curriculum, while reading performance may still be so dependent on the literary and cultural influences in a student's home that it's more difficult for schools to help them improve in that area.

For example, you can see all of the 2003 Reading scores for fourth-graders here. Note that it's good news for the Sunshine State - the increased emphasis on reading skills seems to be paying off for their fourth-graders. The percentage of Florida's fourth graders who are performing at or above proficient reading levels made a significant jump from 27 percent to 32 percent. No other state made a jump that high (Massachusetts, unfortunately, posted a significant decline, although that state's percentage of At or Above Proficient fourth-grade readers (40%, down from 47% last year) is still higher than in Florida.) Connecticut has the highest percentage of fourth-graders in this group (43%), while a dismal 10% of Washington DC youth meet this standard.

If you're interested in the mathematics performance, here are bar graphs showing each state's results for fourth grade and eighth grade. Connecticut, the high scorer on fourth-grade reading, also has the best numbers in both grades on math; 32% of all fourth-graders, and 34% of all eighth-graders, scored at or above the Proficient level.
If you live in Washington DC, though, be prepared to be depressed when you look at these math scores.

Some things to remember about NAEP:

* NAEP test takers are chosen to be representative samples of students in certain grades or at certain ages in public and nonpublic schools in the United States. Thus, not every student will take the NAEP.

* Federal law requires that all states receiving Title I funds must participate in NAEP reading and mathematics assessments at fourth and eighth grades. Similarly, school districts that receive Title I funds and are selected for the NAEP sample are also required to participate in NAEP reading and mathematics assessments at fourth and eighth grades. For everyone else, participation is voluntary, and identifying student information is never disclosed.

* Currently, NAEP reports data only on those samples for which accommodations for disabled test takers were allowed.

* If you're interested in other areas that NAEP assesses, they've got plenty of publications.

* Want newspaper articles about NAEP reports in other states? Here you go: Colorado, Indiana, Maryland, Michigan, New Jersey, Pennsylvania, and South Carolina.

Special accommodations and time limits on the SAT

Hey, John of Discriminations has been blogging up a storm this week, and he ran across the recent NYTimes article on the accommodated testing flagging controversy, which he mentioned in passing in another discussion of a GRE peculiarity (I left comments on that topic, if you're interested).

Anyway, if you'd like to read more about the concept of flagging accommodated scores, and the controversy surrounding the College Board's decision to stop flagging on the SAT, you can go read three of my earlier posts:

Tilting the SAT playing field

Woo woo! Two new SAT

Why did I get my hopes up?

So what's in this latest NYT article?

Last year, when the College Board announced that as of this fall it would no longer flag the SAT scores of students with disabilities who took the test with extended time, educators expected a flood of requests from savvy parents eager to secure every advantage for their children.

Ok, I'll admit that I predicted that, too, and in this posting, I noted that while the percentage of SAT-takers has increased only 18% since 1987, the percent of those requesting accommodations has increased by more than 300%, as shown by the graphs accompanying this article.

Apparently, though, the expected flood ran up against a well-built dam:

From July 1 to Sept. 30 this year, the board received 17,920 requests for extended time and other accommodations, a 10 percent decline from the 19,970 filed in the same period last year.

At the same time, the board has been turning down more requests for accommodations — and the number of appeals from such rejections have more than tripled — in part because of a new requirement that students seeking extra time must generally have a diagnosis and a plan for accommodations in school at least four months before taking the SAT.

So, ETS put sensible procedures in place to turn away those who might be trying to game the system, and as news of that has trickled out, the requests for accommodations has declined. Interesting. In case you were wondering, those 17,920 requests equal somewhere around 1.7% of the SAT-taking population

The board also compiled a list of 142 schools — 43 private and 99 public — where an unusually high proportion of students use accommodations and asked them for further documentation of the disabilities. While those schools represent less than 1 percent of the nation's high schools, they account for 24 percent of all accommodations nationwide.

Faced with such scrutiny, many of the schools that had asked for the most accommodations have pulled back substantially on their requests.

I bet they have. What a smart use of data exploration that was. While some of these schools may have legitimately attracted a larger number of disabled students, I'm sure there were some in there that were taking advantage of the system. Not surprisingly, though, the cries that the College Board is now "too restrictive" has begun.

About 2 percent of the two million students who take the SAT receive accommodations for their disabilities, the majority of them students with learning disabilities who are allowed extra time. The percentage has more than doubled since 1990, amid a troubling inequity: Affluent students are far more likely than poor ones to have documented disabilities and therefore to receive accommodations.

Indeed, some refer to the greatly-increasing learning disability diagnoses as "boutique diagnoses," for which rich familes go "diagnosis shopping."

Historically, the College Board has relied largely on the recommendations of individual schools in granting testing accommodations. In most public school districts, the process is well-defined...The situation is far murkier in private schools, where, typically, parents who believe their child would do better with extra time go to a private evaluator and come back with a report recommending extended time on tests, a report that is usually accepted.

"We have high-powered, savvy parents, and if they come in with a $3,000 evaluation, dead set on getting extra time, it's very difficult to turn them down," said a learning specialist at one selective New York City private school. "I think the College Board's doing the right thing, and helping us not buckle to parental pressure. But right now we're seeing a lot of freaked-out parents."

Just think about those "high-powered" parents who are so eager to have their kids labeled as disabled. These parents have $3000 to spend, and yet they can't manage to find tutors for their kids so that they can learn to take the SAT under normal timing conditions? Am I alone in finding it odd that in one generation, we have swung from the label of "disabled" being a stigma to that same label being something that is seen as desirable, and sought after? Do these parents really have so little respect for the test that they're willing to essentially help their kids cheat? Or are they so desperate to think of their kids as "special" that any means of setting them apart from the general population will do?

At a College Board forum on accommodations in Manhattan, a California educational consultant said on Monday that parents often had trouble accepting that even if an evaluation concluded that their child could benefit from extra testing time, that was not a diagnosis of a learning disability.

"Everyone has strengths and weaknesses, and the fact that someone could benefit from extra time does not mean that they need extra time to level the playing field," said the consultant, Jane McClure.

Gooooood point. Miriam Freedman suggested that one way to level the playing field might be to increase the time limit for all the test takers, simply because most high-stakes exams are somewhat speeded, and many non-disabled test takers could benefit from extra time. Indeed, the NYT story notes that some educators believe that all SAT-takers should have the option of choosing extra time, or that the test should essentially be untimed.

The problem with that choice option is that everyone will choose the most amount of time possible, so that effectively sets a new time limit. A very lengthy time limit - or none at all - can create financial and administrative nightmares. Testing costs go up when seat times lengthen (if testing locations charge for their use), so we could move from a situation in which disability disagnoses are too expensive to a situation in which testing costs become prohibitive. This could also make test scheduling more problematic, and once the time limit expands, the College Board goes from speededness concerns to fatigue effect concerns. Oh, and the new test would need to undergo rigorous reliability and validity analyses because it cannot be assumed that items pretested under one time limit, and administered under another, will remain the same in terms of discrimination, difficulty, and predictive validity. The SAT administered under double time, or no time limit, is a new test, and much data would be needed before colleges could use it to predict first-year performance.

And speaking of validity analyses, those who think the SAT timing should be expanded for everyone are making two assumptions that may very well be incorrect. The first assumption is that increasing the time for everyone would "level the playing field." Problem is, there's no research to show that only students with true LDs would benefit, while others would not, and logically, it makes sense to conclude, as Jane McClure does above, that some non-LD students will benefit from extra time. It is not unreasonable to imagine a situation in which students with true LDs will be more likely to have lower scores on an untimed test, because the non-LD students will have leaped ahead of them due to their ability to benefit more from the extra time.

The second assumption that extended-time proponents are making is that all parents and disability advocates really want a level SAT playing field. No matter how generous a time limit might be, there will always be someone who will ask for more, because there will always be those who want preferential treatment for themselves or their children, as opposed to equal treatment for everyone. If extending the time limit for all in fact widened the score gap between LD students and non-LD students, a lot of people would be furious; there would be no way then to remove the "stigma" of that lower score.

Mixed MEAP results mean less free money

Fewer Michigan seniors will be earning state college scholarships this year, due to a dip in MEAP scores on reading, writing, and math:

The mixed results meant only 51 percent of 2003 high school graduates earned $2,500 scholarships, compared to 54 percent of 2002 graduates. Students automatically qualify if they pass the reading, writing, math and science tests, but also can qualify in other ways.

Problem is, the aggregate scores were released late, and the school districts were frustrated.

Individually, students learned whether they won scholarships by early September, but aggregate scores weren't distributed in time to help school districts improve lesson plans because of delays caused by technical problems with the testing contractor, said Martin Ackley, a spokesman for the Michigan Department of Education....

"This was an odd day for the state to release them," said Diane Blain, a spokeswoman for the 13,000-student Chippewa Valley Schools, which scored near state averages. "It was the deadline for schools (to offer corrections) on the scores. So you had some people trying to get online to make changes, and others trying to read their scores and the Web site was tied up."

Previous posting on MEAP-related headaches can be found here, here, and here.

Another comment about the new SAT

Michael of The Calico Cat blog - which has nothing to do with cats - wonders why a new SAT has been developed. As far as he's concerned, there was nothing wrong with the old one:

I take a break from writing about oil to write about the SAT. As you may be aware, the SAT was the subject of a cover article from Time magazine two weeks ago (yes, I know I'm a little late on this, but better late than never)...

Click here for my take on the aforementioned Time article.

It seems to me that the old SAT was working just fine, why do we need a new one? Of course there does seem to be a lot of people out there who dislike the SAT, but their reasons have never seemed very easy to fathom...

We need standardized testing, otherwise there would be no way to compare students from different high schools...Without standardized tests, elite colleges would wind up only taking students from elite high schools that they were familiar with, and students from high schools that were unfamiliar to the elite colleges would be at a severe disadvantage. Thanks to the SAT, a kid at a mediocre public high school can demonstrate that he's as equally qualified as a kid at a private prep school...

Many psychometricians have made this point. Few reporters have listened.

The new SAT will look less like an IQ test and more like a regular high school test. But there's no evidence that this will make the test more useful for selecting college students...

It's not as if people with high SAT scores and no other qualifications are sneaking undeservedly into Harvard and Yale. In fact, the opposite is true. A kid with a high SAT score but nothing else going for him (no extra-curricular activities and mediocre high school grades) will probably be rejected by Harvard and Yale.

I made that very same point in a response to an article a couple of weeks ago. Yet, the myth persists that people with rotten grades nonetheless get admitted to elite schools in huge numbers just because of high standardized test scores.

I find the graded essay to be the most worrisome part of the new SAT. Essays are much more difficult to grade than multiple choice questions. The cost of grading the SAT will increase, and the reliability will decrease. And college bound kids, instead of spending countless hours practicing analogies, will spend countless hours learning how to write in a way that the SAT graders will appreciate.

Dude. Thank you for repeating what I have said many a time. Of course, some might argue that kids learning to write in ways SAT graders will appreciate is an improvement over the current situation, but never mind. It's nice to see someone else point out the pitfalls of performance assessments.

Kids who want to do well on the new SAT will have to spend countless hours learning the "correct" way to write an SAT essay. I suspect that kids at non-college prep oriented high schools will be at a bigger disadvantage on the new SAT than on the old SAT. And the Time article suggests the same thing.

Those kids are currently at a disadvantage now anyway, because those kids are already less likely to learn how to write, period. At least now there will be more impetus for change in low-performing high schools that hope to send more kids to college. At least the new SAT sends the message that writing skills are important, and as I said before, I don't think it's by definition a bad thing that the SAT requirements might change what gets taught in schools.

Michael speculates that all of this is to try to help close the achievement gap, and he doesn't believe it will work. I said before that the new SAT might introduce more noise into the measurement, and might widen the score gap to boot. Will schools still be willing to stand behind the test if that's what happens?

In a perfect world, there's no need for accountability

From Charleston (WV), here's a personal essay in the Sunday Gazette-Mail, written by a professor of education who questions the usefulness of standardized tests, accountability, and education in general:

Few who have written on the subject [of standardized testing and school accountability] question the assumption that high-stakes testing equals true reform.

Really? I see people questioning it all the time. If anything, the media often seems inclined towards the assumption that standardized tests are flawed, biased, unfair, or unnecessary. If few writers had anything negative to say about testing, I never would have begun this blog.

House Speaker Bob Kiss, in fact, wrote that “Democratic lawmakers and educators agree wholeheartedly” with the conceptual foundation of the No Child Left Behind act. Perhaps he associates with educators with whom I am unacquainted. Many of my colleagues and graduate students, teachers and administrators themselves question the wisdom of substituting “standards-based accountability” for genuine educational improvement.

How many of her colleagues and graduate students have been able to show why standards-based accountability cannot go hand-in-hand with genuine educational improvement? Is there any reason to assume that it cannot? And notice that parents are not mentioned here among those who might be questioning the wisdom of accountability. I don't think it's a coincidence that they were left out of this discussion by the author.

First, as the Gazette’s Oct. 12 editorial recognizes, it is nonacademic factors that best explain the variance among test scores when schools or districts are compared.

Analyses of state tests have found comparable results. Those findings are no surprise to educators, who are familiar with more than 30 years of research confirming the data.

The author presents this as an argument against standards, but couldn't it also be read as an argument against education itself? If 89 percent of the variance between schools is explainable by the background factors, why educate children at all? Doesn't the reliance on this statistic suggest that a child's educational attainment is all-but-predetermined at birth? It seems odd that a professor of education would use this statistic.

I've been all over the Education Week and NAEP websites, and I can't find any reference to a study showing that those demographic variables explain such a huge amount of variance. Regardless, these alleged results only apply to the math portion of the NAEP norm-referenced exam. What about the reading portion, or all the other areas of the exam?

What's more, some would argue that it is bad schooling that has kept children in bondage to those "non-academic" factors not under their control. As E.D. Hirsch puts it, the children from poor backgrounds are more dependent on the quality of their schooling than the children from more privileged backgrounds. To say that children cannot be helped simply because their parents did not go to college is to abdicate any responsibility to help those children who need it the most.

Second, norm-referenced tests were never meant to evaluate the quality of either teaching or learning.

The primary objective of these tests, as researcher and author Alfie Kohn points out, is “to rank, not to rate; to spread out the scores, not to gauge the quality of a given student or school.” To use them in that fashion is not only unfair, but dangerous, particularly in light of the sanctions which can be leveled when a student or school fails to perform “up to standard.”

Here, I have to say that while I believe NAEP is a useful mechanism for comparing schools, one can certainly argue that schools should be measured with the use of criterion-referenced tests instead of norm-referenced tests (for more discussion of these types of tests, click here). However, until everyone in the nation can agree on what those criteria are, there's no way to compare schools across the nation to one another without using the norm-referenced exams that are taken by students in every state.

It's a bit of a dilemma, but it is by no means proof that tests such as NAEP cannot be used to compare performance across schools. It just means that to come up with a system of accountability that is standards-based requires that we develop those standards, and that hasn't yet been accomplished.

Speaker Kiss and others were quite specific on this matter in a Jan. 6 column, noting that the Legislature’s understanding of “accountability” involves “a system of sanctions and rewards.” Sanctions and rewards, however, are not opposite concepts. They are, in fact, quite similar.

One leads students and schools to ask what will happen if they don’t perform; the other, what they’ll get if they do. Like the proverbial carrot and stick, both elicit only temporary compliance, and neither encourages what’s surely a more appropriate question: Is there any real learning going on here?

Is there any reason to assume that there is not? Rewards and sanctions are used to teach children a great deal in their early years, and such methods do not cheapen the learning. Life itself involves sanctions and rewards, but simply because a reward is offered for good academic performance does not imply that the performance is artificial, and just for the sake of the reward.

It isn’t standards that are in short supply here. It’s common sense.

Many advocates of high-stakes testing have not merely ignored but contemptuously dismissed the relevance of the points raised above. Explanations about the impact of socioeconomic status or the failure of standardized tests to measure genuine student learning are written off as “excuses.” This is both disingenuous and thoughtless, and, like any other attempt to diminish the relevance of circumstances, ultimately serves the interests of those fortunate enough not to face them.

I disagree. The assumption being made by the author here is that people who think that poor children can scale great academic heights, despite having the SES deck stacked against them, are not only unrealistic, but elitist to boot. Why assume that all those in favor of standards-based accountability are "fortunate enough" not to face the circumstances? Some of those most in favor of accountability are called "parents," and they do indeed face the circumstances, mainly when confronted with schools who believe that a low level of performance is all that can be expected of children from a certain demographic.

It can be argued that high-stakes tests have precisely the opposite effect their advocates claim. The movement driven by what Kiss described as “an aggressive measurement and accountability system” essentially lowers meaningful expectations. Schools are encouraged to use a one-dimensional tool to measure student achievement.

That's because such one-dimensional tools are often the most reliable and most useful for accountability. That doesn't limit schools to teaching in a one-dimensional sense, however.

Growing disparities among our children are not so much neutralized by public schools as they are reflected in them. Until we insist on more equitable funding for schools, on decent salaries, and on more support for social services designed to help those in need; until we can ensure decent housing and access to adequate health care; and until we establish job opportunities for those who have none, performance standards for large numbers of children will be inaccessible and immaterial. And our schools, as Geoffrey Rips notes, “as the last large mediating institutions in our society, will continue to mediate inequity.”

Ah, so that's is what all this bleating is about. The author is just angry that these demographic differences exist in the first place. So, to recap, this professor - of education, mind you - is declaring public schools to be absolutely useless at educating any students who comes from a poor background, such schools being completely at the mercy of the raw material with which they are working (i.e., the students). It will be only when everyone is born equal, and there's an unlimited amount of money to give to those who don't have any, and we can guarantee jobs for everyone, that we can reasonably expect schools to leave no children behind.

In other words, when cradle-to-grave socialism is successfully instituted in the US, or when pigs fly. Whichever comes first.

One approach to the new SAT in SC

Hilton Head Island (SC) schools are gearing up to help their students perform well on the new SAT. The article is chock-full of useful information for parents:

Because of the wide-reaching effect of the change, Hilton Head High School already is preparing students for the new format of the test...All teachers are including a reading or writing activity in each class everyday...Teachers select activities that are based on the content of the class...All students also have access to a computerized independent study program called "Skills Tutor"...

Students who can't afford the test fee can apply for a waiver...

Busy busy busy

There's no way on God's green earth that I'll get to blog much today; things are too busy around here. Lots of interesting articles out there, though; I'll link to as many as I can, but with more concise commentary than I usually manage.

-------------------

The decision to implement a campus-wide standardized exam at SUNY is drawing even more fire. In June, the SUNY Board of Trustees in June voted for these assessments, but campus faculty senates on 14 campuses have passed resolutions opposing them.

The proposed assessment is intended to "measure student achievement in math, communication, critical thinking, information management, and understanding of methods scientists and social scientists use to explore phenomena." Critics say this exam is a waste of limited funds.

-------------------

Will issues with the flawed MEAP process in Michigan lead to a big revision in the test? Some superintendents say the MEAP isn't a good fit for NCLB because its standards are too tough. Could a national standardized test be the answer?

-------------------

Florida thinks it has found a solution to the delayed-scores problem that plagued the MEAP - computers will be used to score FCAT essays. Not surprisingly, one critic calls this a "nightmare" of "speculative technology"; such critics are apparently unaware that this technology has been one of the most thoroughly-researched topics in testing over the past few years. ETS has one of the more well-known essay raters, e-rater®, but other testing companies have been busily developing their own. Of course, things can still go wrong with computerized essay scoring, but enough has gone right so far that it's premature to label this a disaster in the making.

However, computerized (or online) testing is not easy, nor is it cheap, and it may be the inevitable security or scheduling or cost issues that makes an online FCAT unworkable. I doubt the computerized essay scoring segment of the assessment would be the real stumbling block.

-------------------

Some parents in Berkeley (CA) are unhappy with recent test scores at Rosa Parks Elementary School. All the other schools in Berkeley are meeting standards, but this is the fourth year that Rosa Parks has failed to do so. The district must now create a plan to overhaul the school, which is moving further away from the state target scores while other schools move closer.

In response, the parents claim that the school is not failing their children, but the standardized testing program is. Much of what the parents have to say, though, contradicts any theory that the test is the problem:

“The community is very motivated here,” said Cathy Duenas, the mother of a fifth grader, who said she was more concerned about the tests themselves than the student’s performances. She echoed several parents who expressed concerns about a district-wide trend toward larger class size, citing her son’s math class, which has one teacher for 37 students.

And this is a criticism of the test...how? If the school isn't providing adequate education, then the school, which also did not begin its required English tutoring program until February of this year, is to blame.

Extending the use of the SAT

What's a testing opponent's worst nightmare? The use of the SAT by employers after a potential employee has graduated from college. The Volokh Conspiracy notes that the Wall Street Journal has reported on this new trend; some employers are allegedly requiring the submission of SAT scores along with a college transcript for entry-level positions.

I don't have a subscription to the online version so I can't reach the original article, but here's what The Volokh Conspiracy - and one reader - had to say:

SATs For Life? My colleague Lloyd Cohen passes along these thoughts:
------------------
A most interesting article in yesterday's Wall Street Journal. Apparently, there is an increasing trend among employers to ask applicants for their SATs long after graduation from college. I find this heartening. The SAT is fundamentally an IQ test. While not the only measure of likely productivity on the job, intelligence is probably the most powerful and robust predictor. In the past college of attendance, major, and grades, while always subject to unreliability, were more powerful indices of both intelligence and other productive inputs than they are now. As these other predictors have become more debased it is good to see that the market is responding.
--------------
Precisely because of its value as an IQ test there are those who wish to transform the SAT into something else, and wish to narrow its employment. One wonders how--if they succeeed--the market will respond.

Of course, the SAT does have some limitations as an IQ test, not least that taking a preparation class can boost results, and that language skills for the Verbal section will be affected by the quality of English spoken in one's home. But in the push to raise high school graduation rates and college attendance rates, graduating from high school or even attending college is no guarantee that a job applicant has basic literacy and math skills, much less advanced ones. It's not surprising that employers are looking for substitute signals.

Not surprising, but that doesn't mean that the SAT is necessarily the best test for the job. It is related to intelligence, and it does predict first-year college grades. However, that doesn't mean that it is necessarily predictive of success on the job once one has a college degree, given that most of those with college degrees will have a more limited range of SAT scores (limiting the range of a variable necessarily limits the correlation of that variable with other variables). Any company wanting to use SAT score as any sort of selection variable would do well to collect data showing that SAT scores were in fact correlated with productivity, or performance reviews, or some such measure of job "success" in that environment.

What's more, if employers plan to use the test for hiring or promotion purposes, they're going to come under fire from the same people who use the score gap as a means to bash objective college standards. Colleges already feel the pressure to lower standards for minorities, allegedly because the test is "biased." Companies who use the SAT will most likely see the same score gaps (it would be interesting to know if they didn't), and they'll suffer the same pressure to not use the test because of those gaps. I wonder if they have any idea of the abuse they'll suffer if they turn down someone with a decent college GPA who had a low SAT score and was admitted to college under affirmative action.

Did anyone see the original article, or have a subscription to the online version?

Update: Apparently it's not in the online version. One helpful reader who did read the article mentioned a standard of 1350 as the SAT score for at least one company. I'd be very surprised if companies were using scores this high as absolute cutoffs. As this chart shows, a 1350 is at the 93rd percentile. Any company using that cutoff is almost certainly ruling out some qualified applicants.

What's more, take a look at this chart. It doesn't give composite percentile ranks, but we can easily come up with an example that illustrates the problem. Let's say the most likely combination of Verbal and Math that produces a 1350 score is 700 Verbal, and 650 Math. Let's compare the percentiles by score and race:

5% of White examinees score above a 700 in Verbal, while only 1% or so of black or Hispanic examinees do so.

19% of white male examinees score above a 650 in Math, and 11% of white females do. However, only 2% of blacks and 4% of Hispanics do this.

See the problem? The score gap exists throughout the SAT score scale, but at the high end of the scale, the number of black and Hispanics examinees drop off dramatically. Any company that requires a 1350 SAT score for hiring purposes is in effect doing what the College Board tells colleges not to do, which is the use of a strict cutoff score so high that it will eliminate a large proportion of minorities. Companies that use this high a cutoff won't have to waste much money on "diversity training", because they won't have that diverse an employee population.

Companies are, of course, perfectly free to make the decision to use the SAT, but I hope they're prepared for the lawsuits to come.

The importance of parents

Pulitzer-Prize-winning journalist Clarence Page recounts the amusing tale of a talk-show host who was desperate to find someone who disagreed with Abigail Thernstrom and the theory that black parents do not do enough to convey the importance of education to their children:

As much as I love to argue, especially in front of vast national television audiences, I had to bow out when a popular cable TV talk show recently asked me to debate author Abigail Thernstrom on the delicate topic of the academic achievement gap between black and white students.

There was a brief moment of silence on the other end of the line. I felt my shot at the national limelight evaporate and they would not be interested in booking my son. That's show biz.

Mr. Page then goes on to scold such media folk for trying to stir up controversy and argument around this topic, when calm, level-headed discussion is much more necessary:

One of the most disturbing disappointments in the years since the 1960s civil rights revolution is that the black-white academic performance gap (as much as four years by the time they graduate high school) persists, even among children of the new black middle class...

Yet, whites are not the top performing group. As the Thernstroms point out, the gap between white and Asian-American student performance is actually wider than the gap between blacks and whites, with Hispanics performing about as poorly as blacks.

Emphasis mine - I've always opposed AA because I believe it masks the real problem, and I believe that researchers such as the Thernstroms are doing a good job of showing us where the real issue lies.

One important idea that the Thernstroms are publicizing is the innovative "trouble threshold" theory; it is, in essence, that the students who do well are the ones with parents who do not let grades go very low before introducing punishment (I remember getting skinned and fried for anything less than a B). The Thernstroms cite a 1996 study which surveyed students about parental reactions to grades. White kids said they had to stay above a B- to avoid trouble, whereas black kids could earn a C or C- before the parental wrath would descend.

Asian students? Anything less than an A- and they'd be grounded, and that was the same for both immigrant and native-born students. What's more, Asian parents were more likely to believe that "academic performance depended entirely on how hard [the student] worked," and much less likely to believe in innate abilities, good luck, and other such matters that weren't under the control of the student - or the parent.

This isn't the only difference the Thernstroms uncovered - for starters, young white kids were often reared in homes that had, on average, twice the number of books as young black kids - but it's hard to deny the data that show the impact of parental attitude on academic performance.

Wonder if that talk show host is still looking for someone to debate Abigail Thernstrom? That person better come prepared with some serious data, and a suit of armor to boot.

Educating New Hampshire's parents

I found an interesting little article from the Rockingham News Online. The article describes a forum given by school officials to educate parents on standardized tests and other assessments. Definitely an admirable service to provide (I hope they began by explaining how to pronounce New Hampshire's tongue-twisting test acronym, NHEIAP.)

A comment near the end of the article caught my eye:

It is also important not only to focus on the test scores but to challenge the students in new ways.

"These things are often more important that test scores," said Hume-Howard. "There has to be a balance between testing and the classroom."

I just thought the description of these factors seemed oddly out of place. Is it really the business of schools to teach compassion? And are compassion and "learning style" really more important than test scores for college applications these days? I have no problem with schools focusing on work ethic, nor on them helping children fulfill their leadership potential, but the way those qualities are presented here, you'd think they were completely independent of test performance. A child with a solid work ethic is going to perform up to their potential on any assessment, including standardized exams.

What's "up" with scores in San Francisco?

San Francisco school districts are reporting a jump in test scores. Is that due to increased focus on students whose second language is English - or does the change in tests explain the increase?

Ask school superintendents and principals why standardized test scores increased so much this year, and they will credit increased focus on helping students who do not speak English. But a change in the way the scores are calculated may also have something to do with the improvements.

Panorama's scores had been dropping five years ago, Presta said. Administrators found that non-native English speakers were scoring poorly, and the school hired a specially trained teacher to help. They also increased individual student assessment, especially in reading. "We really believe that it's the reading," she said. "The kids weren't reading at the level they should be."

This article describes how the API works:

According to the Academic Performance Index, an annual survey of standardized test scores at elementary, middle and high schools throughout California, 90 percent of schools showed an overall improvement in their scores during the academic year 2002-2003 over the previous year.

In San Francisco, 86 percent of all schools met their growth targets during the school year 2002-2003, according to district officials, who were in a jovial mood at a news conference Friday morning to announce the results.

The same effect is being seen all over California, which is not surprising. Increased district scores in Monterey are described here. The California Department of Education has all the API results here.

Here's the information for parents (requires Adobe Acrobat). This documents points out that the CAT replacements for the Stanford 9 are specifically aligned to California content standards, but the entire STAR system allows California's students to be compared with each other, and with students in other states.

Acrimony over AIMS in Arizona

Not much is chilly in sunny Arizona, but state schools chief Tom Horne felt the chill when he proposed combining Arizona's AIMS with the Stanford 9 exam. He believes Arizona's students are over-tested, and claims that his plan will save money, save time, and be ready by 2005. Apparently, some board members feel AIMS needs to be expanded, not shortened; others feel the public doesn't accept the current AIMS, and that expanding or modifying it will not help.

To MEAP, or not to MEAP?

A couple of weeks back, it looked like Michigan's MEAP was on the way out. The MEAP is expensive, and the score gap is substantial. But now it appears that Michigan's educators aren't quite ready to replace the MEAP, despite its flaws, and spoke to members of the House education committee about the exam:

Educators attending a meeting yesterday to discuss the future of the Michigan Educational Assessment Program acknowledged the state’s standardized test was imperfect, but they weren’t ready to give up on it...Six state representatives from the House education committee listened as more than a dozen educators from throughout southeast Michigan expressed both their pleasure and concerns with the test...

Overwhelmingly, the nearly 50 educators and community members at yesterday’s meeting said yes...

Why the "overwhelming" response? Apparently, because districts have put a lot of time, effort, and money into revamping classroom curricula to match the MEAP. Changing the test means changing the class content as well. However, the educators did suggest some changes:

...they offered suggestions on how to make things better, namely not to make the test count for so much when dolling out the districts’ progress reports. Also, educators expressed concerns about how special education students, transient students, and low-income students factor into their scores.

Dissing the exams

Someone at the Lumberjack Online (which appears to be the student paper of Northern Arizona University) is NOT happy about all these standardized exams:

...the situational tests life throws in our path are not always the ones determining how far we get on the educational spectrum.

In fact, the scores a student receives on a single standardized test at the college level can, at times, determine the difference between being admitted to the college of their dreams, or settling for mediocrity.

Not to sound unsympathetic, but I wasn't aware that all kids were guaranteed the right to be admitted to "the college of their dreams." Why is the only other alternative listed as "settling for mediocrity"? What's wrong with "settling for pretty good"?

As a matter of fact, ETS and The College Board, along with every other producer of admissions exams, do not advise colleges to use the score as a cutscore in admissions. A score band is always provided that allows for the error of measurement, and schools are perfectly free to give the tests as much, or as little, weight as they choose. Some schools don't use the SAT at all.

Just as SAT scores can determine scholarship eligibility and undergraduate admittance, most graduate schools require students to take the Graduate Record Examinations prior to acceptance into a desired program. If one is not happy with their score, electing to take the test again is an option, but it costs a pretty penny.

The test costs $115 (page 3), or two months worth of giving up cable TV. Fee waivers are available for students with limited resources (page 7). No, the test is not free, but the cost is not insanely prohibitive, either.

GREs are not the only standardized tests required for graduate school. If you plan on attending law school, you had better be prepared to take the LSAT, and you better do really well. According to www.lsatcenter.com, most law schools consider your scores on the LSAT to be more important than your accumulative GPA. The same is true for admission to medical school or any other type of esteemed specialized program.

And the reason for this is not because the schools worship standardized exams, or because they like to torture students. It's because LSAT scores predict first-year law school grades better than GPA alone, and since virturally all first-year students within a law school take the same coursework, the schools want the best way to choose the students who will live through that rough first year. Note that the schools make the decision on how to weight the LSAT, instead of LSAC making that decision. If the tests were not useful, the law schools would not use them (yes, I'm aware that accredited US law schools must require the LSAT, but they can weight it any way they like).

Personally, I think graduate schools need to re-evaluate their admission criterion. All things considered, most students agree that standardized testing does not accurately measure a person’s intelligence.

And they're not supposed to. They're supposed to predict who can handle coursework during the first year. Certainly, students with test anxiety may do poorly, but such students often have a slew of accomplishments in college to back up their claim of readiness, and schools are perfectly free to give more weight to grades than to test scores. Schools that do not give much weight to GPA may very well be the school of one's dreams, but when almost every kid applying has a GPA above 3.8, how is the school supposed to be able to distinguish the truly good from the grade-inflated? That's where an objective measure comes in.

After all, most people have to pay big bucks for graduate school, so why would admissions counselors let one score on a test affect a student’s admission status?

Because schools can't take every student who can afford to pay. Is the author suggesting that they should? Should ability to pay be the element that is more important than test scores? That hardly seems fair.

Every year, billions of dollars are spent by students who are trying to make themselves stand out by scoring well on required standardized tests. I have no idea where that money goes, but somehow I don’t think it’s put toward a worthy cause.

Um, it goes towards the test prep companies, for starters. Those are for-profit companies that use the money to produce more books and more advertising, and occasionally to sponsor scholarships. It also goes to the test developing companies like ETS and LSAC, which are not-for-profit companies that use the money to pay psychometricians to perform research showing that the tests are as valid and reliable and useful as possible. Test development companies also spend money on test prep and diagnostic services(page 8).

What would this writer consider a "worthy cause," I wonder? If the schools suddenly decided not to use our services, I suppose we'd all be out of business, but as long as schools (a) cannot accept all applicants and (b) need a quick and objective way to select those applicants who are best qualified for their programs, the tests will be necessary.

We deserve to know if our money is being used to better our educational systems, or if it’s being used to make an elite group of money-grubbing businessmen even richer than they already are.

Ex-cuse me? If you're referring to the for-profit test prep companies, go right ahead, but psychometricians who are "money-grubbing" are in the wrong field. None of us are getting rich off of this, and money isn't why we entered the field in the first place. There are a few for-profit testing companies springing up, but because they are mainly hired by school districts, they're not exactly raking in the bucks, either. ETS and the College Board and LSAC are, as I mentioned, not-for-profit, which means their employees do not get rich off testing fees.

I see these kind of articles all the time, really. They're always written by people who are convinced that tests just appear from on high, like Moses' scrolls, and who have no interest in understanding the financial or manpower needs that are required to keep such tests going. One form for an average large-scale, high-stakes, paper-and-pencil exam probably costs around $500,000 to develop, but that cost goes towards making sure that the quality assurance procedures, psychometric staffing, item writers, data entry folks, techies, and production values are as superior as possible. No one is making money off these exams; we spend the money to produce the exams; to score the exams; to figure out ways to improve the exams.

I find it hard to believe a doctor could get C’s all through his undergraduate career, ace one measly test, get admitted to a top-notch medical school, and still be allowed to cut open patients.

That is hard to believe, indeed, because it's not true. Who has convinced this writer that an applicant with poor undergraduate grades and a high MCAT score will get into Harvard Med, much less master Harvard's curriculum and then pass the medical board exams? That's ridiculous. Schools have a name for this type of applicant - "underachievers." These are students with potential who, so far, haven't displayed much of it. Competitive schools aren't going to admit this type of applicant.

Someone with a high GPA and a low MCAT score probably won't get into Harvard Med, unless there's a slew of other mitigating factors and/or Harvard has decided to downplay the MCAT for certain applicants.

However, with entrance exams weighed as heavily as they are, I am sure that kind of thing happens quite frequently.

No, it doesn't. As I pointed out above, the combination of motivated students and grade inflation practically ensures that almost all applicants to top-notch schools will have high GPAs; the low GPA students are the ones who don't stand a chance.

I propose one simple solution: All standardized admittance tests should be abolished. Intelligence is ambiguous. Nobody will ever be able to evaluate how smart someone is, so testing administrators need to quit stealing people’s money, and look at what really matters – work ethic, grades and a heck of a lot of self-determination.

Oh, that's so dumb, and a complete cop-out. This writer, citing no evidence to support her claim, completely ignores all the decades of accumulated data about intelligence, apititude, and ability testing, and demands that schools admit all determined students who work hard and make good grades. Never mind that "work ethic" and "self-determination" are more difficult to judge than intelligence. Never mind that at some colleges, almost half of all students have A averages, which doesn't leave post-collegiate programs much in terms of decision-making. Never mind that tests such as the GRE and SAT are equated over time, while grades are not. Never mind that there is some evidence supporting the claim that high school grades may be inflated, although not to as large an extent as many claim. And if intelligence is so "ambiguous," why admit only those students with good grades? Couldn't it be argued that those with low GPAs were "intelligent" in ways that couldn't be measured in class?

I'm not sure why I wasted time on this article, given that the writer does nothing more than use ad hominem attacks and straw-man arguments to convey the extreme pissy displeasure of someone who still hasn't gotten over getting a low SAT score. C'mon, people. There are legimate testing criticisms to be made; it's sad to see a writer with access to a school newspaper waste space with tripe like this.

"Get Out Of Detention Free" card

Students at Destrehan High School (LA) recently received their School Performance Score cards, or SPSs. An SPS tallies the points that a student received based on his or her standardized test performances and attendance record. The twist is that it seems to be like a little credit card, where privileges can be purchased with the points:

The card has a maximum of 24 points, which can be used for privileges.

A student also may skip a lunch detention or be excused from a tardy with the use of one SPS point. Future SPS card days will be dress down days.

Seems to me that the school is essentially setting in stone what was once an informal set of rules, where teachers were more likely to be lenient with the "good" kids who worked hard and only screwed up every once in a while. I'd have loved to have a card getting me out of a detention based on my test scores, I'll tell you that much.

Masking the Gap

Well, well, well, a columnist for the Collegiate Times has apparently had it up to here with all this talk of SAT bias (I can empathize, believe me):

During the early 20th century, rich white males who went to prestigious and expensive prep schools dominated college admissions. Implementation of a nationwide standardized test was thought to be the best way to give everyone an equal chance of getting into college and receiving scholarships.

Great care goes into making sure the test is fair and balanced...If this test is biased, it is only biased against those with no math or English skills.

Heh. So why the gap?

Grade inflation is rampant in U.S. high schools across the country. With close to 40 percent of high school students graduating with at least an A- average, how can schools distinguish between candidates without another attribute?...

Another argument against the SAT is scores do not indicate college success. Women do worse on the SAT but yet have better grades in college. However, more women take classes that are arguably easier, such as marketing and communication...On top of that, courses in engineering and science have not been subject to the same skyrocketing grade averages.

True - the hard sciences have been less politicized and subject to less grade inflation. I won't say that there's none, and certainly professors can grade easier by giving points for extra credit, but math and science departments tend to be sanctuaries for the politically incorrect, and when the division of right-vs.-wrong answer tends to be clear, it's harder to argue the grading scheme.

Any study that compares college grades for different subgroups, without taking college major into account, is essentially producing greatly confounded results. One blogger (whose name I've forgotten) recently listed 10 college majors, ranked by difficulty of the course work; if I recall correctly, physical education was ranked above Women's Studies, or Gay/Lesbian Studies, or some such major where the only right answer depends on the professor's sex and political party affiliation.

The columnist here, Kris Hassinger, floats the theory that women and minorities do more poorly on the SAT simply because lower-ability women and minorities are less likely to find good job opportunities that would be available to lower-ability white males, such as mechanic, construction worker, etc. (Please don't write me letters telling me about all the genius mechanics you know. I'm not saying they're dumb, but they probably ended up in their fields because they were smart in ways other than "book-learning.")

Therefore, the SAT means are lower for women and minorities because they don't feel that they can do the dirty blue-collar work, and they search for another means of employment, most of which involve college degrees.

Kris winds up by dissing UC's decision to downplay the SAT I, and makes an observation that I myself have made on here many a time before:

The University of California is pushing to throw out the SAT in admissions in an effort to bolster diversity...Something is fishy if a student has a 4.0 but gets an 800 on the SAT, and it’s not the test. Let’s not throw out academic data just because it is inconvenient to our sleight-of-hand effort to increase minority acceptance.

The underlying problem is the dismal school systems and societal de-emphasis on educational merit. Blaming the test for the poor scores is essentially like shooting the messenger.

Yes, it most certainly is. We psychometricians strap on our metaphorical bulletproof vests every day, in fact; it helps us in dealing with the misguided and the mercenary who oppose any sort of objective testing.

TIME's take on the new SATs

The new issue of Time has the goods on the "dramatic" changes on the New SAT. At first, I feared that the editors had gone with the "gloom-n-doom" approach, because the fact that some students' scores could be "hurt" is mentioned in the subtitle, and the article begins with:

Three hours of misery are apparently not enough. Now the makers of the SAT want to shape what kids learn throughout four years of high school. True, students have always had to brush up on vocabulary and take practice tests before the SAT, but now the College Entrance Examination Board, which owns the test, is developing the "New SAT," an exhaustive revision largely intended to mold the U.S. secondary-school system to its liking.

Please tell me that this isn't setting the tone of the entire article Why, only one paragraph in, and the College Board has already taken over the entire K-12 system.

The College Board wants schools to produce better writers, so the New SAT will require an essay. The board thinks grammar is important, so the new test will ask students to fix poorly deployed gerunds and such. To encourage earlier advanced-math instruction, the New SAT will go beyond basic algebra and geometry for the first time to include Algebra II class material (remember negative exponents—q?3, for instance?). The board, a powerful group of 4,300 educational institutions—including most of America's leading universities—has undertaken an unprecedented effort to push local school districts to alter their curriculums accordingly.

I take it all the Time's editors and writers send their kids to schools where writing skills and grammar are unimportant, and no math beyond Algebra II is considered useful?

...Instead of the venerable math and verbal sections, the test will have three segments that will be more familiar to Americans: the three Rs, reading, writing and arithmetic. (Hence a perfect score will go from 1600 to 2400.) At first blush, the changes seem healthy enough. But inevitably, some students will do better, and some worse, on the new test.

Why is that newsworthy? Is it because the assumption here is that the only fair test is one on which all subgroups perform equally well? Do the writers at Time really believe that is possible? Or am I just being hypersensitive, thanks to the spate of uninformed and unsupported claims of test "bias" recently?

Girls tend to outperform boys on writing exams, so their overall scores could benefit from the addition of the new writing section. Boys usually score higher on the math section, but the new exam will contain fewer of the abstract-reasoning items at which they often excel. The elimination of analogies may exacerbate the black-white SAT score gap, since the gap is somewhat smaller on the analogy section than on the test as a whole, according to Jay Rosner, executive director of the Princeton Review Foundation.

Oh, jeez. Here we go, again. Does this mean that even Time writers don't see the vast flaws in Mr. Rosner's reasoning, nor the ability to wonder how there might be a conflict of interest here? If Mr. Rosner can scare more kids into taking prep courses, allegedly because he has inside information about "biases," shouldn't a journalist wonder aloud whether that sort of information is good for business?

More broadly, students who attend failing schools could suffer as the SAT morphs from a test of general-reasoning abilities into a test of what kids learn in school. "There's a danger that making it too curriculum-dependent will actually increase overall score gaps for some minority groups," says Rebecca Zwick, a former chair of the College Board's own SAT Committee. "Because we have such huge disparities in the quality of schooling in the country, kids who go to crummy schools may be disadvantaged."

Kudos to Time for interviewing an actual psychometrician who makes a very good point. Those who go to crummy schools will be disadvantaged if the test becomes too curriculum-specific. But are the universities concerned about this? Or are they too fed up with the influx of diploma-holders who need remedial math and English courses to care? Perhaps the only students they want are the ones who can show that they've learned something in high school.

As Time itself points out, this sort of change in the test can drive the the curricula that schools use. Unlike Time, I don't immediately assume that this is a bad thing.

The world of standardized testing has its own language and history; an entire branch of science, psychometrics, is devoted to test design and analysis. But the tiny discipline touches most Americans' lives at some point...

Hee hee hee. You just don't know how much I appreciate that phrase, "tiny discipline." Not only is it funny, it's true - there aren't many of us in the world (and some are very thankful for that, I'm sure).

Insta-experts from the media and from antitesting groups often repeat fallacies: blacks do better in college than their SAT scores predict (actually, for reasons that aren't well understood, blacks tend to do worse in college than matched groups of whites with the same scores); how well you do on the SAT will determine how well you do in life (SAT scores have little power to predict earnings).

Hooray. Finally, an article willing to call a fallacy a fallacy, although there are plenty more they could have listed here (and certainly the names Rosner and Freedle should have been mentioned as well).

Six months ago, TIME asked the College Board if we could sort out some of these conundrums by following the development of the New SAT from inside. To our surprise, board president Gaston Caperton III agreed...

At his insistence, the goal of influencing school curriculums has become the overriding preoccupation of the new test's developers...To be sure, Caperton believes the notion (actually, he's staking his career on it) that the SAT can both improve high schools and still remain useful to colleges as a predictor. But the first goal is a political aim; the second, a psychometric one. And Caperton has surrounded the New SAT with dozens of educators who aren't schooled in psychometrics.

Reeeallly. Now that's interesting, and good reporting.

For decades, the SAT was, at its heart, an aptitude test; now it's becoming more like its competitor, the act, the nation's biggest achievement test. What's the difference? Achievement tests gauge mastery of subject matter; your U.S. history final was an achievement test...

Whereas early psychometricians, many of them racist, propagated what Lemann calls the dipstick theory—the idea that a test score is like a mark on a dipstick showing the raw amount of intelligence in your mental oil tank—the field outgrew that simplistic notion at least a generation ago.

Um, more than that, unless a generation is fifty years or so. But hey, at least they're not still calling psychometricians racists; many reporters haven't made it that far. And their explanation of the distinction between aptitude and achievement is very useful for readers.

...the more you challenge yourself intellectually, the more you condition your brain; your academic achievements are less impressive if you don't have the conditioning to build upon them. As the SAT becomes more an assessment of one's achievements, it will less sensitively gauge these underlying skills...

The use of technical language will also increase in math. For instance, in the past, an SAT item might have stipulated that group A has 10 members and group B has 10 + 5x members, where x = 3...on the New SAT, the question might read, "What is the union of sets A and B?"...The answer is still 35, but you must know the jargon to get it.

This is a fine point, and I'm glad the Time writers took the path of questioning what could happen with the shift towards more technical language and jargon, and away from pure symbols, instead of immediately assuming that this change is unecessary, or unfair.

I put that question to David Lohman, a University of Iowa psychology professor who has studied the differences between achievement and aptitude tests. In a paper that will be published in the forthcoming book Rethinking the SAT , Lohman analyzed test scores for 6,300 11th-graders who in 2000 took two very different tests, the Iowa Tests of Educational Development (ited) and the Cognitive Abilities Test (CogAT)...

Others have replicated such findings by comparing achievement and reasoning tests in earlier grades; one theory as to why minorities often score higher on the latter is that they attend poor schools that leave their potential untapped. "Indeed," writes Lohman in Rethinking the SAT, "the problem with the current version of the SAT"—which continues to show a racial score gap—"may not be that it is an aptitude test, but that it is not enough of an aptitude test."

Very interesting. Time also points out that it has been schools such as the University of California that have demanded this change, rather than the College Board. So the new SAT, which may be more likely to measure the "higher-order thinking skills" that educrats so favor, may also exacerbate the score gap, and it may do by measuring achievements that students at poor schools are less likely to attain, regardless of their innate ability.

If the subject-based SAT IIs are doing just as good a job of predicting performance as the SAT at some schools, that suggests that those higher-order skills may be what admissions tests should be measuring - even if the subgroup differences widen. But some claim that isn't the case. What does it mean if a combination of high school GPA, SAT, and the SAT II is the best predictor? What do we conclude about what better prepares a student for college? Is reasoning, as measured by something like the CogAt, really enough?

The paradoxes, thorny decisions, and Catch-22s (now there's some literary jargon for you) here are really something. The new SAT reading passages will come from actual works of literature (although I assume they will bowdlerized to some extent), meaning that those passages won't be deathly dry and dull anymore - but this also means that students who have already seen those passages may very well have an advantage. It's not that it's bad to encourage kids to read those books, but Time is correct in surmising that this introduces noise into the measurement of reading comprehension.

The more performance assessments a test contains, the more noise the measurement contains, and the new SAT essay will be no different:

As Lemann writes of the early rationale for the SAT, "Tests that require a student to write essays ... are highly susceptible to the subjective judgment of the grader and to the mood of the taker on the day of the test, so they have low reliability."

Reliability is a measure of a test's precision from one administration to the next—a gauge of how much noise, or measurement error, it has eliminated. The standard error of measurement for a typical SAT is about 30 points for the math section and 30 for the verbal...Thirty points in either direction is a pretty big swing, but scores on the writing section will be even less reliable: field trials of the New SAT estimate a standard error of measurement of 41 points. That means a kid who gets a 670 may "really" be in the élite reaches of the 700s—or in the more average environs of the low 600s.

Grading essays is not an easy task. Setting standards so that raters can be trained to efficiently and accurately rate essays is very difficult. The College Board has their work cut out for them. Combining the essay score with the multiple-choice score mitigates the noise problem somewhat, but not entirely.

Today Atkinson and Caperton have launched another great social experiment with the SAT. This time, the idea is that the test's rigorous new curricular demands will lift all boats—that all schools will improve because they want their students to do well on the test. Schools have long tried to prepare kids for the SAT, but education experts scorned the practice of openly teaching to the test. Now it's the mission of the College Board that every school should teach to the SAT. "I would say that the most important aspect of this test is sending a real message back to kids on how to prepare for college," says Atkinson. It's not clear what happens to students in schools that won't hear, or can't afford to heed, his message.

Not a bad ending. And not a bad article; in fact, it was much better than I thought it was going to be, given the tone of the introductory grafs. An important question has been raised. Are those educators who so often devalue the "basic skills" and "lower-order thinking" that the old/current SAT allegedly measures going to support this more difficult test, even if the score gaps widen? Certainly, the gaps should be a wakeup call to struggling schools, but will those schools claim that they cannot "afford to heed" the message?

I don't see how they can afford not to.

The growing popularity of the PSAT

Some schools in Delaware are now requiring ninth- and tenth-grade students to take the PSAT, and this seems to be part of a larger, nationwide trend:

Delaware is joining several states nationwide this year by requiring ninth- and 10th-graders to take the exam, typically taken by 11th-graders. Taking the PSAT has been voluntary until this year...

The College Board can analyze results of the PSAT to identify students who might be overlooked for advanced classes.

Some students are already nervous about the exam, but the results don't count until the 11th grade, when good scores can bring cold hard cash in the form of National Merit Scholarships. What's more, the PSAT is excellent practice for the SAT, as well as being a useful tool for identifying student abilities, and I agree with one administrator that free PSATs are a "gift" to the students.

The Montgomery County School District in Maryland has given the PSAT to its 10th-graders since 2000. Spokeswoman Kate Harrison said 67 percent of its high school students are enrolled this year in at least one honors or advanced placement class, up from 55 percent in 1999. The percentage of black students in the district enrolled in advanced classes went up from 31.7 percent in 1999 to 47.4 percent in 2002. Hispanic enrollment in advanced classes rose from 31.6 percent to 42.4 percent and Asian enrollment from 69.5 percent to 81.1 percent during that time period.

I think it's a great idea, and it's nice to see some data that suggest this type of preparation can help all students.

Missing the point about the SAT score gap, again

In today's Hoya (the Georgetown University paper), we find yet another well-meaning, yet naive, article about efforts to close the SAT score gap between subgroups, most particularly whites and minorities:

...Recent statistics show that the minority SAT gap persists: minority students generally score 150-200 points lower on average than their white counterpart, according to the College Board. Statistics released by District of Columbia public schools indicate that the gap is particularly prevalent in the Washington, D.C. area. In 2003, black students’ average combined scores were 438 points below that of whites’, and Hispanics’ average scores were 446 points lower than white students’...

Dennis Williams, the director of Georgetown’s Center for Minority Education Affairs, criticizes the SAT, emphasizing “the scores measure family income more precisely than they measure how well you can do in college.”

No, the scores measure how well you are prepared for college, which is related to how much money your parents have, because more money often means better schools, more tutoring - and more educated parents to start with. Ms. Williams needs a refresher course on correlation coefficients, and how a correlation between X and Y may appear because both are correlated with Z.

And even if SAT scores were correlated just with income per se, and not with the opportunities for educational accomplishment that extra dollars afford, does Ms. Williams provide any reason for us to assume that students with low SAT scores/low incomes will do well when placed in college? No, she does not, but SAT critics rarely provide any of this type of supportive reasoning.

Parental involvement also plays a large role, Williams says. Parents who did not themselves go to college — while not less involved in the lives of their children — are less likely to understand the college admissions process. Educated parents are in a position to help their children more, says Williams.

Again, what about this is supposed to convince me that the SAT should be removed, so that kids who have not accomplished as much, be it due to parental involvement or income, will have a greater shot at college? College isn't a babysitting service that should be open to those with the most need, as opposed to those with the most potential.

Another argument involves some of the SAT questions themselves. Although Educational Testing Services, which produces the test, claims it is trying to eliminate any bias inherent in the exam, Jay Rosner disagrees...“The test company uses a completely neutral, colorblind system for picking questions. However, that system predictably, consistently and reliably yields questions that favor whites dramatically over other subgroups,” Rosner says. The result of these rules is that the only questions that end up being on the SAT are “white-preference questions,” he says, which are questions prescreened and chosen to be on the SAT that white students answered correctly more often than other students.

You can go here to read my take on this blather, or you can simply ask yourself a few simple questions that go far towards demolishing his arguments. Do you really believe that all 2000 or so of the ETS and College Board employees are devoted to making sure black students cannot get SAT items right? For what reason would they do this, other than that they believe only white people should be going to college? Do you really think the KKK is running ETS? And if you believe that ETS and the College Board have the power and will to construct the test in this way, why do Asians outscore whites on the math portion of the exam? Don't you think the KKK would control for that, too? What about that "Other" group that's doing so well? Is that just because ETS can't purposely create items that are biased by race if examinees don't tell ETS what their race is?

You cannot go far with the likes of Mssrs. Rosner and Freedle without having to accept as fact the implied statement that ETS and the College Board are engaged in a deliberate conspiracy to foil black college hopefuls. If you really believe that, I know more than a few psychometricians who would be happy to help you readjust your tinfoil hat.

While he acknowledges that part of the discrepancy is in test preparation, Rosner asserts that if they let him pick the questions, “there would be a dramatic close in the gap.”

And if you let me tinker with your bathroom scale, there will be a dramatic change in your BMI measurement - but you won't be any skinnier. Yet another writer has missed the chance to point out that closing the gap at the end does nothing of value for minority students, because they're really suffering from an achievement gap, not a score gap.

Minorities may also face difficulty in preparing for the SAT. How well a student performs seems to be related to how familiar they are with the test; both Kaplan Inc. and The Princeton Review Inc. preach that the test is teachable. Their courses, however, are often priced close to $1,000 — a hefty fee for students in low- and middle-income families.

And this reflects badly on the SAT...how? The College Board didn't set those prices. Another way to look at this is to say that Kaplan and the Princeton Review are profiteering from the struggles of minority children trapped in bad schools. Test prep is virtually free for any child motivated enough to look for it.

Of course, these programs are garnering huzzahs for reducing their prices for minority students, so I suppose no reporter would dream of taking these companies to task for setting their prices so high to begin with. It's much easier for everyone just to blame the test.

What follows is some interesting information about foundations and efforts to increase minority performance on the test:

In Washington, D.C., specifically, the Hoop Dreams Scholarship foundation raises money to help students succeed through a series of three-on-three basketball tournaments. The Princeton Review provides courses for underprivileged students in conjunction with this program free of charge. A program called “Keys for Life” is also run through a partnership between The Princeton Review and the Child and Family Services of D.C.

Through the Good Sports program, Margaret Arbuthnot raised her SAT scores to a 1420 and is now a freshman at Princeton. She says she is grateful for the services the program provided but that she was motivated to succeed on her own. “If I had to pay a lot for it, I wouldn’t have done it,” she says. “I didn’t know how important SAT scores were in admissions because I’m the first kid in my family to go to college. I recommend test prep, but I think it’s outrageous that people have to pay a thousand dollars for it.”

The author, Lindsay Breedlove, also points out a recent College Board study showing that retaking the exam (which is cheaper than a prep course) tends to help students prepare better, and raise scores higher, than do the expensive prep courses. Unfortunately, she then follows up with the Ridiculous Professor Berube and his fantastically idiotic golf-handicapping suggestion, which is presented here with a completely straight face.

So, Rosner, Freedle, and Berube, along with other testing critics, are given space in this article without so much as a single quote from a College Board employee in response. This sort of imbalance exposes the ideology/gullibility of the writer and keeps this article, which does convey some good information about scholarship programs, from being entirely believable.

Update: Ms. Breedlove should take a look at this editorial that urges readers to take minority test results "seriously;" this means skipping over the pat answers about score differences and getting to the heart of fixing the educational issues:

Most teachers will tell you students learn in different ways and at different rates. To look for common indicators may be appropriate in view of the national learning gap, but they also run the risk of degenerating into stereotypes that do more to excuse poor performance than to correct it.

Bridging the gap between white and minority students must be an essential goal of quality education. It must be approached critically and objectively.

And not with loopy conspiracy theories or inane athletic analogies.

Bad timing in Cedar Rapids

Some Jewish parents in Iowa are upset because a standardized test was recently scheduled to begin on a major Jewish holiday, and they're wondering why the school is making a mistake that's that equivalent with requiring Christian students to test on Christmas:

Keeping school events from conflicting with religious activities can be quite a challenge. But when a recent standardized test began on a major Jewish holiday, it upset some parents. That's why Mary Cooper decided to address the Cedar Rapids School Board this week. She feels the district needs to be more sensitive to religious holidays. Other members of the Jewish community agree...

Feller says this is not the first time the Jewish community has had a concern. They do try to work with the schools to avoid this kind of problem. "We've written a letter every single year and enclosed a calendar and let people know. But perhaps we're not letting the right people know," says Feller.

Indeed, the school should be as aware of this as possible. When an organization has the money to create different tests to be given to students on different days in order to accommodate religious beliefs, that's the ideal solution. But most schools don't have the money for multiple tests, and I understand that picking one school day that's suitable for everyone can be trying. Perhaps schools should simply select the three most important holidays from several major religious as no-testing days, and declare every other day fair game?

Rising enrollment, falling test scores?

One Illinois school district says rising enrollment is causing their test scores to fall:

Rising enrollment is hurting the standardized test scores for students in St. Charles schools, district officials said Monday. Test scores for elementary, middle and high schools show that while students are exceeding state standards in every subject, there were few major increases or declines.

She said that while teachers have been hired to handle the enrollment increases, the budget for training and other teacher support has stayed the same.

So they're still way above average in test scores for the state, and didn't see large declines in scores. Why is the training budget key here? Is the assumption that those 600 new teachers don't know what they're doing, and thus aren't able to make all the test scores improve?

Reporting on scores the right way

After all my complaints about reporters whose articles reflecta complete misunderstanding of what test score gaps mean, it's nice to see a good example of a testing-related article that addresses the issue reasonably and gives the readers useful information:

The 2003 scores on [Georgia's] most crucial exam [the Criterion-Referenced Competency Test or CRCT] are reason for optimism and concern.

However, thousands of kids still aren't at grade level when tested on core subjects, and the state's black and Hispanic students still are lagging well behind white students.

First thing to note: Overall, students are improving, but the kids who aren't improving should be a concern. It is also a point of concern that certain ethnic subgroups are performing worse than others.

"The good news is everything improved, and that is really positive," state Superintendent of Schools Kathy Cox said. "What you see when you disaggregate [by race] is you're still dealing with an achievement gap."

Bingo. This is one person who has it exactly right. These high-stakes testing sitations are indeed helping the student body as a whole. The students who don't rise are lagging behind because of their failure to achieve, not because the tests are biased, or because the expectations are too high. The fact that this administrator specifically refers to an achievement gap, instead of a score gap, indicates that she knows the scores indicate where problems are, and are not in and of themselves the problem.

The Criterion-Referenced Competency Test...[is]...designed to determine whether students are performing at grade level in core subjects...The bump in reading scores is not an accident. Millions of state and federal dollars have been used around Georgia to boost reading performance...For instance, at Shoal Creek Elementary School in Rockdale County, students have 90 minutes of uninterrupted reading time. There are no phone calls, no announcements, no conferences.

"That's a sacred reading time," said Janet Wheeler, now in her third year as principal of Shoal Creek. The program started last year, and Shoal Creek did show a slight improvement in its reading scores. But more importantly, Wheeler said there is a buzz over reading in her school.

Great. The school looked at the test scores and figured out an educational change that might improve scores, and attitudes. The school didn't just decide to spend a lot more money on test preparation, which is a trap that some fall into because, again, they're seeing the scores, and not the skills behind them, as the problem.

That same emphasis hasn't been put on English and math, and it is showing in the results...Performance on the math curriculum test generally lags behind other subjects, although there were gains in all grades. The state is currently revamping its core curriculum...the new curriculum will be more focused and allow more in-depth instruction in core subjects, like math.

The test scores show that many metro Atlanta school systems were among the top performers in the state, although some saw a drop in the percentage of kids who met the state standards. The results also confirm the existence of an achievement gap in Georgia.

Again, I'm impressed by the fact that this reporter isn't shying away from calling the gap what it is - an achievement gap.

For instance, in fourth grade, 88 percent of Georgia's white students met the state standard in reading. However, the number dropped to 73 percent for African-Americans and 65 percent for Hispanics. The gap exists in all subject areas in all three grades.

Experts say poverty, access to resources and socialization are among the reasons that such gaps exist.

It's nice to see "experts" quoted (even if briefly and without identification) on the topic of root causes instead of the standard critics who disagree with testing in general. Many reporters believe, I think, that test results - especially score gap results - cannot be discussed in a "balanced" fashion unless comments about test bias (usually broad, unsupported anti-testing claims) are included as well. That's not true, not least because one rarely sees testing critics define "bias" correctly or provide data to support their claims.

SUNY's standardization

Escaping the K-12 system doesn't mean escaping from standardized exams. The State University of New York (SUNY) system has begun a pilot implementation of a campus-wide standardized test to see how its approximately 411,000 students are doing at all 64 locations:

A pilot program for the test is set to begin next fall and run for two years. There is no firm starting date for the full-scale version of this test.

Relying on essays and other nonmultiple choice questions, it is unlikely to be a graduation requirement. Instead, it is viewed as a way to help gauge the effectiveness of various programs throughout the SUNY system.

The stakes of this test are thus low for students, but higher for the campuses and programs involved. The question, then, is, how useful are the data if the students feel no pressure to perform well on - or even be present for - the exams? Sometimes, no-stakes data can give good information, but such data always need to be taken with a grain of salt.

That much said, I'm betting that the complaints we'll see will parallel the complaints of K-12 schools that are rated. Campuses with low test scores will claim that the test is biased, or inappropriate for their students, or that it contains inaccurate or misaligned content.

The more basic the skills on the exam, though, the less valid the claims become. It's hard to argue that a test of basic English and math skills is too standardized (or "cookie-cutter") for any college campus. If the community college scores, for example, show that, after two years, students still haven't mastered these skills, that would be a pretty devastating piece of information suggesting that students are wasting their money at these campuses.

Answering Mathews' "Bias Question"

John Rosenberg bopped me on the head this afternoon with a reminder that the "Bias!" crowd is still at it. The latest installment? An article by Jay Mathews in The Atlantic online entitled,"The Bias Question", in which Mr. Mathews swallows the former ETS employee Roy Freedle's suggested SAT revisions hook, line, and sinker.

There's a lot of interesting backstory here, mostly about Dr. Freedle's research experience at ETS and his interest in linguistics, which formed the basis for his research and the Harvard Educational Review article, "Correcting the SAT's Ethnic and Social-Class Bias: A Method for Reestimating SAT Scores."

John dispatched this article rather quickly, by noting that Mathews had concluded with the following fallacy:

[According to test critics] The problem [with minority score gaps], in short, is with the test rather than with any reality the test measures.

Now, is it my imagination, or does John seem weary of the repetitive nature of these sorts of test "criticism?" Perhaps I only hear that weariness because I, too, am weary of having to state the same arguments over and over again, while the testing critics shut their eyes and ears. My weariness means that I simply don't want to do a long, in-depth criticism of the Mathews article. Much of what I would have to say, I said before when I criticized the original Freedle coverage in the Chronicle, and when I criticized the bias allegations by another researcher in a more recent Chronicle article.

I suggest you go read the Mathews article for yourself, but there are a few remarks by Mathews that I simply have to address:

Freedle's accusation of racial bias in the SAT is striking because it is one of the few ever to come from an experienced ETS professional...

Striking, perhaps, but why assume that because it is one of the few, it must be right? Or even that it should be given more than a passing glance? I sense here an assumption (which I see from many testing critics) that ETS is an evil place that deliberately manipulates tests so that minorities fail, and no one there is allowed to contradict the "party line." Therefore, if anyone who used to work at ETS makes a statement that contradicts claims by ETS or the College Board, then that person must be right....why? Because what ETS/CB claims cannot be true?

I mean, imagine this statement:

Dr. Smith's insistence that the germ theory of disease is incorrect is striking because it is one of the few ever to come from an experienced medical professional...

Would you conclude just from this statement that Dr. Smith must have stumbled upon something that has escaped all other doctors up until now? Or would you conclude that he's an eccentric and perhaps confused researcher determined to push an agenda that contradicts the established research?

You can take "eccentric" for what you will, but unless the assumption is that the medical community (or ETS) has conspired to "hide the truth", there's no reason to conclude that one researcher from "the inside" who insists that all the established research is wrong, and that we must now start doing everything a different way, is automatically believable.

Perhaps more important, it has caught the attention of the University of California (a powerful malcontent in the College Board family), which has ordered its own detailed analysis of the issue, due to be completed in 2004. Even if Freedle is ultimately proved wrong, his success at raising doubts about the SAT shows how loose a grip the test has on the political and scientific handholds that keep it upright.

That's funny. I would assume that if Freedle's research was proven to be wrong, that would be one more link in the chain of evidence that the SAT can be a useful tool for predicting first-year college performance for all test-takers. Again, the assumption here seems to be that ETS and the College Board support the SAT only through a conspiracy of hidden research, and the fact that this one person came to light must be proof that the whole house of cards is about to fall down.

I've previously addressed Freedle's suggestion that we tinker with the test solely to decrease the test-score gap without decreasing the education gap, and the College Board has addressed the fallacies of his research. But I still can't let this one portion of Freedle's research, as described by Mathews, go by:

Common words, Freedle explained, "often have many more semantic (dictionary) senses than rare words," so there's more of a chance that people's cultural and socio-economic backgrounds will affect their interpretations of those words. (In a 1990 study Freedle and Kostin reported that "fifteen high-frequency analogy words ... had an average of 5.2 dictionary entries, whereas rare analogy words ... had an average of only 2.0 dictionary entries.") Thus words that are frequently used in the middle-class neighborhoods of the SAT makers may have a different meaning in underprivileged minority neighborhoods. This, Freedle continued, could help explain why African-American students do worse on questions containing those common words than on questions that depend on the harder (but less ambiguous) words they study at school. He found that this effect was most pronounced on those questions—sentence completions, analogies—that provided little or no context.

This is Freedle's argument for why African-American test-takers should be helped on the exam by a revised scoring system which scores only the hard items, and not the easier ones. Leaving out the very large question of why items should be given at all if they're not going to be scored, I have to address the "educational relativism" that's holding up this theory.

So, high-frequency words are more likely to be used in different ways, by different people, in different environments. Fine. However, the dictionary lists word usage in specific orders for a reason, and just because a word can be used in different ways, it doesn't necessarily follow that students cannot be expected to know when a particular usage is correct in a particular situation.

When Mathews, and Freedle, say that "words...used in the middle-class neighborhoods of the SAT makers may have a different meaning in underprivileged minority neighborhoods," what they're both desperately trying to avoid saying is that minority students are getting these meanings wrong on tests. They're trying to avoid admitting that in the context of a test item, just as in the context of a portion of writing, words have right and wrong meanings, and there's no reason to give a pass to a student who doesn't learn how to use words in context, and no reason to give their schools an out for not teaching them that.

For example, let's use the noun "deck". The online Merriam-Webster gives the following definitions:

1 : a platform in a ship serving usually as a structural element and forming the floor for its compartments
2 : something resembling the deck of a ship: as a : a story or tier of a building b : the roadway of a bridge c : a flat floored roofless area adjoining a house d : the lid of the compartment at the rear of the body of an automobile; also : the compartment e : a layer of clouds
3 a : a pack of playing cards b : a packet of narcotics
4 : TAPE DECK
- on deck 1 : ready for duty 2 : next in line : next in turn

So, I say "deck," you might think of the deck behind your house, being on deck for the next task - or a packet of cards, or narcotics. But a test might contain this word in an analogy item, such as the one I just made up and don't claim is perfect:

"Deck is to ship as foundation is to (a) house (b) sky (c) bird (d) cow"

The deck of a ship is essentially the structural element which serves as the floor, so it's analogous to the foundation of a house (Like I said, this isn't perfect. Don't write me emails telling me everything that's wrong with this item).

This item has little context, but the only thing required of the student is that, at some point, the student has learned the primary meaning of the word. Nothing in Freedle's research convinces me that we cannot expect schools to teach all students the multiple meanings of words, and therefore should be willing to change the test scoring to make life easier for students who have not learned how to use words in correct ways.

Isn't Freedle's argument essentially that African-American and white students cannot be expected to learn the same definitions for a set of words within an educational environment, and therefore we (as testmakers) cannot insist that students understand all uses of a word and in which contexts they should be used? If our educational system is short-changing African-American students to the point that they never learn how to use common words in all the ways that they're used in proper written English, the test-score gap is one of the few meaningful indicators of this. Why aren't African-American students learning the same meanings of words as white students, and why should we modify the test to make accommodations for this, rather than demand that schools do their jobs and give all kids as extensive an education in reading comprehension and the English language as possible?

The fundamentally dishonest idea of closing the test-score gap at the end (test results) rather than at the beginning (the teaching of students) pervades this article, and Mathews' conclusion is breathtaking in its wrong-headedness:

Nearly all those involved—ETS and College Board officials, University of California researchers, high school guidance counselors and admissions officers from those schools that would be affected by a change in the SAT—are, like Freedle, practical people with a seemingly distant but still compelling goal. They want to remove barriers that limit young people's choices in life. All of them, Freedle included, acknowledge that many other things, more difficult than devising a scoring supplement to a multiple-choice test, will have to be done to make that happen.

I've emphasized the above because I've said this so many times before, and even though I'm sure I'll be saying it again soon, I want it to be clear. Tests are not barriers. Tests measure skills, and if students do not gain those skills, they are by definition barred from achievement whether the tests are in place or not.

If the items are re-weighted and re-scored so that the score gaps closes, this hides the fact that African-American students are actually learning less in school. Freedle's suggestions do not ensure that everyone will go forth (with their "different" understandings of the English language) with the same chance of succeeding in school and in life simply because the "barrier" of the test has been removed.

The barrier is bad education, not the tests that measure educational achievement. Why do reasonable, logical writers like Mathews miss this point time and time again?

Channel One's take on the SAT

A Devoted Reader who is also a teacher suggested I visit the Channel One site to view an SAT video. I can't get the video to come up (even though I signed up for Channel One membership); luckily, my reader gave me a synopsis of it (which I have edited for clarity):

Today Channel one showed an article on the SAT. "Is the SAT an indicator of college success?"

Bob Schaeffer from FairTest is interviewed. [why am I not surprised?]

Complains about the "underlying issues of the accuracy and the fairness
of these (SAT and ACT ) tests." [none of which are ever defined or supported with data in these types of reports]

A 16-year-old student, "Some people aren't good at taking certain kinds of
tests." Another 16-year-old pro test comments. (Channel one usually presents one student opinion from each side.)

Reporter: "Critics say SATs are biased against minorities [critics that rarely provide data or use the correct definition of bias, that is.]

Bizarrely, the video then shows black and Asian students [who generally outperform whites on the exam], and students from lower income families.

Bob comes back to say gaps between Blacks, Latinos, and Anglos have grown. Calls the SAT a strategic game. [What does he call the K-12 system that underprepares these students?]

My reader also had this to say:

I teach math at "A Channel One School". The last three years I taught Statistics. This year I'm teaching Advanced Math (Pre-Calculus) and AP
Calculus. My school is 75% Black in rural NE North Carolina. The reaction to the article in my Advanced Math class was not suprising. The vocal students were the struggling blacks. They wanted the SAT eliminated. My top students (Black and White) said nothing.

Comments by those types of "vocal students" are often presented as legitimate test criticism by clueless reporters. While the SAT has differential impact on different groups, that doesn't necessarily mean the test scores are less valid for blacks than for whites, and the fact that some students struggle with the exam is no indicator of its fairness, or validity.

Revising score weights in the Grand Canyon State

Arizona's State Board of Education has revised its method of grading schools by giving extra emphasis to Measure of Academic Progress (MAP) scores. Interestingly, this is supposed to help schools that might have been labeled as "underperforming" by the previous rating system:

"Underperforming" schools can expect visits from the Arizona Department of Education to assess where deficiencies exist and to help establish a plan for improvement. Schools that fail to shed the negative label for three years could possibly be taken over by the State Board of Education.

To get the full 40 percent toward the label, a school will need to have at least 90 percent of students achieve one-year's growth. Schools with MAP scores of 70 percent or more will get 20 percent of the label credit and schools with 60 percent or more will get 10 percent. Schools with less than 60 percent of students achieving one-year's growth will receive no points in the labeling equation.

Information on the MAP scores can be found here, with information on Arizona's standards here.

Test Anxiety in Alabama

Mobile County (AL) students are almost done with the administration of the newest standardized test, the Criteria Reference Tests, which are intended to spot academic problems early on. The tests also contribute to students' grades, which may explain the pessimistic tack of this article:

...it will be another week before Mobile County school officials are able to review the tests to determine whether the questions were fair and the results accurate, said Lee Taylor, the system's assistant superintendent for curriculum.

Some said the students were stressed because the test counted for so much of their quarterly grades, Goodwin said. Others complained that because the tests didn't match the county's curriculum, they were instructed to throw out or give the students answers to several questions.

Stress about the high-stakes really isn't newsworthy, but misaligned content matter, tossing items, or providing answers are indeed "glitches" that must be addressed. At least one principal was enthusiastic about the process, but many teachers felt that this first round of testing didn't go very well.

Regents on the right track?

Last week, the New York Board of Regents voted to hold back, for at least two years, the implementation of new, higher standards for all students:

New York State's education commissioner, Richard P. Mills, said Wednesday that the state would loosen the demanding testing requirements it has imposed for high school graduation in recent years, including the standards used to judge math proficiency...

Mr. Mills said he would give special education students two more years before they had to pass Regents exams to get their diplomas. Until now, they have been allowed to graduate by passing less rigorous tests known as the Regents competency tests. That test had been scheduled to end in 2005.

Now, the NYTimes is questioning the wisdom of any implementation of one set of standards for all:

The board's most recent actions — prompted by an outcry over faulty state Regents tests — illustrate some of the difficulties attached to such an approach, some more easily solved than others.

An independent panel examining the state's Math A exam in June, which 63 percent of the students failed, concluded that the test itself was badly flawed. They said that if the state uses "make or break" tests, then it must spend the money to get them right.

Conceded. Mistakes will always happen as long as humans are involved, but the higher the stakes, the more that quality control must be continually and carefully applied.

But critics of the standards-based approach say there are other problems, too. Some students will never reach the same academic level, and the one-size-fits-all approach is just not fair, they say.

"The question is, How are these kids who end up leaving school without a diploma — either because they have failed a test several times or because they have been pushed out — better off?" said Robert A. Schaeffer, public education director of FairTest, an advocacy group that is critical of standardized testing. "And how is society better off if they can't get a job because they don't have a diploma?"

Whatever. FairTest trots out these pat statements any time a testing scandal arises. The question that should be asked is, how is society better off when our K-12 system can't educate kids well enought to pass basic skills exams in high school? Yes, the Regents exams were flawed, - but many other state exit exams are not, and the percentage of 12th-graders passing 10th-grade-level items on those exams is often astonishingly low. FairTest seems to think that the diploma in and of itself, rather than the classroom experience, confers upon students the skills needed to function well in society.

New York used to issue different diplomas for students at different levels. Only the top students, who took the toughest Regents exams, got the prestigious Regents diploma. It was not until 1996 that the Regents made the exams a condition of high school graduation for everyone.

What tripped up New York was the same thing that is tripping up many other states: it is easy to create standards but hard to make sure that teachers and students can meet them.

Actually, it's not even that easy to create standards, not ones that everyone can agree on. Getting teachers to follow them and students to meet them is even more difficult. That doesn't mean, as FairTest would have you believe, that testing in order to hold kids to these standards is necessarily a bad thing. If the anti-testing crowd worships diplomas so much, I say that NY should figure out a way to bring back the bifurcated diploma. Some kids will pass only the NCLB standards, some the more rigorous Regents diplomas.

But no state will be able to implement standards successfully until two things happen:

(1) Test development, administration, and scoring are implemented as promptly, thoroughly, and error-free as possible, and

(2) People admit that some students do not have what it takes to earn a high school diploma these days. The reactions to this may vary - some may suggest improving K-12 education, some will suggest an alternative vo-tech route, some will say the GED should cover these kids - but until people are willing to accept that exit exams will always prevent some kids from graduating simply because these kids have not mastered high-school-level skills, there's no reason to have exit exams.

Update This just in from Newsday: Two Regents Board members are suggesting that students who fail one Regents exam should be allowed to graduate, as long as the average of their other four exams are above the passing score. Not all the Board members seem happy with this idea, though.

Raising standards - and remedial enrollment?

West Virginia is raising the ACT standards for college students to be able to enroll in for-credit classes in state public universities:

Beginning in 2005, West Virginia will require that entering freshmen achieve an ACT score of 19 in English, up from 18, and 20 in mathematics, up from 19, to make the cut into for-credit classes in those subjects.

According to Mullen, an estimated 10 to 15 percent of students at West Virginia University and 25 percent of students at Marshall University would not have been able to meet the stiffer qualification on the standardized test, he said.

This raise of one point apparently makes quite a difference; around 700 students were admitted with the lower scores last year and would have been placed in remedial classes if the new standard were in place.

The ACT scores on both English and Math range from 1 to 36; a requirement of 19 on English and 20 on math requires that West Virginia's students be at the 46th and 55th percentiles, respectively. Kudos to WVA for raising standards - but it's odd to see a Chancellor admit that students are going to be admitted to the state public universities with the understanding that they will be funnelled directly into remedial, not-for-credit classes. If that's the case, why admit them at all?

The mormons have uncovered the holy grail

The Holy Grail of academic achievement, that is. Salt Lake City (UT) School District officials are claiming that they have helped non-Asian minority students in their school districts improve on the reading and math sections of the Stanford Achievement Test at twice the rate of their white and Asian peers. What's the secret?

The improvement defies state and national trends and is all the more significant given the district's growing cultural diversity...The trend means the district is improving the academic achievement of its most at-risk students without jeopardizing the achievement of others...

Reading scores followed a similar pattern, increasing from 28.7 in 1997 to 34.2 in 2002 for minorities and 53.3 to 54.7 for white and Asian students.

So the gap still exists, but it is closing. The school credits "intensive teacher training" to meet the reading and math needs of minority youth for the change. However, even those scores for the white and Asian students don't look that great to me.

Here's more information on the Stanford 9 Achievement Test. The test compares students against a national cohort of millions of students, and the test is normed so that 50 is the average score. Thus, Utah's white and Asian students are doing slightly better than average, but the non-Asian minority students are still well below average. Given that they began at a lower point, this rapid rate of change may not be as miraculous as Salt Lake Officials claim.

More testing in Mobile

After a 20-year hiatus, the Mobile County (AL) school district has decided to re-introduce quarterly standardized exams that will count for up to 20% of students' grades. The exams will be given starting in kindergarten, and failure on the test could drag a barely passing student down to a failing grade for a class:

The quarterly exams -- known as CRTs or criteria reference tests -- are being introduced this year after an almost 20-year absence. System officials have said the tests will allow teachers to spot student academic troubles early on, rather than letting students slide through various subjects unnoticed until yearly standardized test scores come in.

"We have some teachers who are a little uneasy. We're plowing new ground," said David Thomas, school board member. "But I think this is the most productive measure we have ever taken to assure that all children are receiving a quality education."

The FCAT, again

Florida's juniors and seniors who failed the 10th-grade FCAT are gearing up for one more go at it, and they must pass it in order to graduate:

...Statewide, more than 70,000, or more than 42 percent, of sophomores failed at least one part of the FCAT. This week, those juniors and seniors will get another chance to clear the hurdle when they retake the FCAT...

West attributed this difference to how the 10th-grade exam tests for critical thinking skills, which students gain largely through an activity that is not necessarily popular with many students: lots of reading.

Sounds like it's time to start making reading popular with this crowd, then.

The state recently announced that rather than taking a traditional exam, which includes written responses, students who retake the FCAT will only see only multiple-choice questions to speed the grading process. The change has prompted concern that the state is making the test easier to pass and whether the state is setting up different standards for first-time and repeat test-takers.

Not necessarily. I can understand the need for greater speed, and there's no reason that the multiple-choice items must cover different content, or must be easier, than the open-ended items.

So far, the situation in Florida has prompted emergency measures:

At the end of the 2002-03 school year, of the 12th-graders who still needed to pass the exam, only 31 percent passed the reading section and 42 percent passed the math section, according to state figures. That left about 12,500 high school seniors who were ineligible for a standard diploma.

The law, valid for only one year, allowed students who have scored at least 14 out of 36 on the reading and math parts of the ACT to qualify for graduation; on the SAT, students would need to score 370 (out of 800) on the verbal part and 350 on the math part.

Hey, they could get into Berkeley with those scores!

Seriously, though, the FCAT requirements are starting to sink in for Florida's educators, and all the talk about how some kids don't test well, or about how kids just don't like to read, aren't going to help these kids get by. Some kids have genuine test anxiety, but I don't believe that explains why such larger proportions of the seniors can't pass, with multiple takes, the reading portion.

Another whack at the dead horse

Devoted reader and darn good blogger John Rosenberg forwards me the text of an article from the Chronicle of Higher Education about that favorite dead horse of the anti-testing crowd, the ostensible racial bias of SAT items. I don't have a subscription, but I do have John's email, so I can post the "highlights" and comment on them:

Before a question appears on the SAT, it is carefully pretested by the Educational Testing Service, which administers the college-entrance exam. As a result, test designers know in advance how members of different groups fare on each and every item. On some questions, white test takers consistently pick the right answer, while black and Hispanic students trip up, even though the questions appear to be similar in content and level of difficulty. On other pretested items, however, minority test takers perform consistently better than white test takers.

But because of test-construction rules used by the ETS, which do not take race into account but which are aimed at maximizing the test's reliability, the only questions that end up on the SAT are "white-preference questions," says Jay Rosner, executive director of the Princeton Review Foundation, a nonprofit organization that helps underprivileged students prepare for standardized tests.

On the October 2000 SAT, he says, every one of the 138 questions on the test favored white students, and none of the pretested questions that had favored minority students were included. He says he has found the same bias in the tests in other years, based on extensive data purchased from the testing service.

Horse puckey. Only three paragraphs in, and both Mr. Rosner and the Chronicle are already demonstrating an appalling lack of understanding of how tests are constructed, the definition of bias, and how items are analyzed for differential functioning, or DIF.

Items are removed from the test if they show DIF in pretesting. An item that shows DIF is one on which ability-matched examinees of different subgroups have different probabilities of answering items correctly. The key is the matching, and this key point is what no testing critic or racial bias agitator wants to address.

If, for an item, high-scoring white students have a significantly different probability of answering the item right than similarly high-scoring black/Hispanic/Asian students during the pretesting, that item will not be assembled into the test. It does not matter which group the item favors; the item will be removed from the pool. This is how psychometricians define "bias," and by this definition, the SAT is not biased, because SAT items go through several layers of this review process

Anti-testing activists insist on moving the goalposts and fudging the definitions. Their definition of bias is, did blacks as a whole do worse on this item than whites as a whole? For a test of academic achievement in a country in which black students are more likely to come from poor homes, more likely to attend poor schools, and (thanks to subversive cultural elements), less likely to value academic achievement, by this definition, the items will be biased simply because blacks as a whole ARE doing worse on the items than whites. The test score will have differential impact on blacks and whites, but this is no more inherently unfair than the fact that a driver's license exam will have differential impact on those who know how to drive vs. those who don't.

That nonsense above about "every one of the 138 questions on the test favored white students" is, in fact, nonsense because Mr. Rosner is comparing all black students to all white students, and because white students tend to come from more academically-rigorous schools, a larger proportion of white students are answering each item correctly. He's simply looking at group mean differences.

The problem is that true bias has little to do with group mean differences, and everything to do with the measurement of extraneous constructs. Here's a retro example of such constructs. If a group of male and female students answer a math item which asks about how many yards of tulle and jacquard fabric are needed to make a set of curtains, we might expect to see male-female DIF. We might see that high-scoring males have less chance of answering the item correctly than high-scoring females.

We then remove the item from the test - not because we think it's "unfair" for women to have the advantage, but because the item is measuring something that it wasn't intended to. We don't care if boys or girls know what "jacquard" means; all we care is whether they can calculate square yardage. "Fairness" is not the issue; validity is. An item measuring a nuisance construct is an invalid item.

This is what testing critics refuse to address. On the one hand, they castigate the shortcomings of the public school system for minorities, as well they should. But then they turn around and castigate the black-white score gap on the SAT, which is one of the biggest pieces of evidence proving that minority students do get shafted in the public school system. The SAT test score gap supports a call for K-12 reform. It supports the NAACP claim that black students don't get challenged enough in school, or are too quickly shoved into special education programs. It supports the claim that black students get shortchanged. Why, then, attack the test?

There is no indication in this article that Mr. Rosner has shown that the lower black scores are invalid; he's only shown that they exist. He did not show that the items measure extraneous concepts that are ineffable for even high-scoring black students, which is the only way the items could be biased, despite the insistence of testing critics to use whatever definition of "bias" suits their fancy.

So what exactly is his evidence?

Mr. Rosner is quick to point out that the bias he charges is not intentional on the part of the College Board. "They're not racists," he says. "The test company uses a completely neutral, colorblind system for picking questions. However, that system predictably, consistently, and reliably yields questions that favor whites dramatically over othersubgroups."

How does that happen? Mr. Rosner says that the explanation comes from the way that designers of the SAT -- and of many other standardized tests -- define reliability. A reliable question that is intended to be difficult, for instance, is one on which those who score highest overall consistently do well, and on which those with low scores consistently do poorly, he says. Therefore, the highest achievers among test takers set the standard, and questions on which they do well look more statistically reliable than those on which they do poorly, even though some minority students may score higher on those questions.

"It's entirely internally cyclical and self-reinforcing," says Mr. Rosner.

This doesn't sound like any definition of reliability with which I am familiar, and I have no idea where Mr. Rosner is getting this information. To start with, difficulty and discrimination are two different components of an item, and while they can be related, they don't have to be. A question can be both reliable and easy, which means that the high-scorers aren't "setting the standard." The only way in which high-scorers set the standard is to have a test on which all the items are difficult, or all the easy items are completely unreliable, and neither of these apply to the SAT.

And isn't what Mr. Rosner is saying here is that the College Board should be using a non-color-blind method to assemble the test, and use items that are of lower reliability (by his definition)? Like the hapless Mr. Freedle, also mentioned in this article, Mr. Rosner doesn't want bias eradicated; he just wants more items that favor blacks and a less reliable test so that the score gap will go away without effort. That way, we can all ignore the miserable job our K-12 system is doing of educating minority youth.

Wayne Camera of the College Board has quite a few good replies to both researchers:

Wayne J. Camara, vice president of research and development at the College Board, calls Mr. Rosner's analysis "fairly superficial and unsophisticated."

Mr. Camara agrees that some types of questions show performance differences among different groups, but he describes those as "very small differences between students at that ability level." [Emphasis mine]

And he says those differences can be explained by different ability levels. [Emphasis mine, again] For instance, he says, "on geometry items, males do substantially better than females." But Mr. Camara says that such differences should not lead the testing service to stop testing geometry in the name of fairness. [in other words, group mean differences do not mean bias exists.]

"It's not practical, and it's not psychometrically defensible to privilege ethnic difference over the content," he says.

What about Mr. Rosner's offer to build a fairer test by picking different questions?

"He's using racial- and ethnic-group difference as the primary criterion for assembling a test," says Mr. Camara. "It's not difficult to do that, but the test you end up with is not measuring what you want to measure."

Exactly. Both Mr. Rosner and Mr. Freedle envision a world in which tests are assembled so that both the black and the white test-takers will do exactly the same overall, and if this means that the items are genuinely biased, or unreliable, or that the test measures a lot of noise along with reading comprehension and math skills, so be it. This is what such critics consider to be "fair," in fact, and it highlights just how far away the concept of "fair" has moved from the concepts of "valid" and "color-blind" for the anti-testing agitators in this era of racial over-sensitivity.

SAT study tips

Helpful hints for those about to bury their noses in study guides and sharpen their Number 2 Pencils. It's SAT Prep time again, folks:

SAT Preparation Tips
* Ask for help from everyone you know.
* Get plenty of sleep the night before the test.
* Don't study the night before your test date.
* Pace yourself -- spread your study schedule over a period of at least three weeks.
* The morning of the test, eat a healthy breakfast.
* Get to the testing center early.
* Bring plenty of pencils and a calculator.
* Pace yourself during the test -- answer questions you know right away, save harder questions for later.

To which I must add:
* Work as many practice tests as possible UNDER THE TIME LIMITS. Buy the tests cheaply from the College Board.
* Review all items, so that you understand why you got right answers as well as wrong ones.
* Discount any and all newspaper articles in which "activists" scream about the racial and gender-based "biases" of the exam.
* Discount any and all comments from teachers about how this exam doesn't measure any higher-order thinking, or any blather about how Einstein would have done poorly on the test.
* Bring headphones with soothing music (even if, for you, that means Pantera) to listen to on the ride to the test site, and to listen to during breaks. Whatever you do, don't listen to the hyperanxious students who discuss the test during breaks. They'll only psych you out.

Another SAT prep article can be found here.

The lawsuit rodeo in dallas, texas

An education activist in Dallas is suing his local school district in an attempt to force them to release the Iowa Test of Basic Skills (ITBS) results, among others, at the classroom level. Activist Russell Fish believes these results will show the public how well - or how poorly - certain instructors are teaching. The school district, which vehemently opposes releasing scores, claims that to do so would violate student privacy. Interestingly, Mr. Fish has the NAACP on his side:

Legal experts say that if Mr. Fish's suit is successful, it could be used as a precedent to open up performance data on teachers all over the state.

Mr. Fish, 50, is a retired electrical engineer and volunteer tutor. He also calls himself an "education activist." He says he has been using school district computer records to track public education in Texas for more than a decade...

Mr. Fish claims that the school district fears public knowledge of teacher quality. He also claims that the district's charge that students can be identified from the data is bunkum (and I have a tendency to agree, if the data are released with identifiers rather than names). Teachers, on the other hand, are legally able to be put in the spotlight, according to Mr. Fish's counsel.

What data are we talking about here? The Value-Added Teacher Assessment (usually called VAT), in which teachers are linked with test-score gains made by their students over some time period, usually a couple of years. The theory behind VAT is the opposite of what our hapless Mr. Alford claimed yesterday; VAT states that teacher quality is the critical variable in student educational gains, with race, SES, gender, and so on falling a distant second in the amount of impact.

The article gives a good description of VAT, so I won't go into all of it here. Suffice it to say that schools intended the data to be seen only by administrators and the teachers involved, for the purpose of "internal discussions." Texas releases a lot of other state-, district-, and school-level data, but has traditionally drawn the line at teacher-level data:

The thought of holding teachers publicly accountable for their own classroom test scores summons up a variety of fears among critics who say public schools already place too much emphasis on high-stakes testing. They argue that many variables other than "teacher quality" intrude on test scores and that it's unfair to focus so much attention on the instructor.

"If you are going to publish teacher rankings, then also publish the heroics and beautiful things that teachers do to ennoble our youngsters," said Dr. Erika Karres, an education consultant and former teacher in New York. "You can't consider teachers and students as machines that you can gauge by an input-output industrial model."

Dr. Karres doth protest too much. We know that teachers are more than their test scores; on the other hand, a teacher whose "heroics" leaves students lagging behind when it comes to performance on objective measures is not doing his or her students any favors, no matter how "beautifully" they may be doing it.

The district and Mr. Fish wholeheartedly agree that a couple of years of ineffective teaching can have devastating effects on a child's school performance. Where they part ways is in the decision to hold that knowledge confidential, or release it to the public. Be interesting to see how this turns out.

Mr. Fish has a theory. He says public education used to be "a sacred trust" run by very smart women whose career opportunities were limited to teaching, nursing and a few other professions. "They firmly believed in their bones that they had been placed on earth to impart knowledge to boys and girls," he says.

Clayton Trotter, Mr. Fish's attorney...argues that even-handed public policy should require teachers to be treated the same as police officers, firefighters and other publicly funded professionals whose personnel records are mostly classified as open records. In Texas, state law specifically prohibits public disclosure of a teacher's annual performance review.

Testing vocational students

Should vocational students who fail the MCAS nevertheless be allowed to graduate? Some advocates think so and, along with groups such as the Massachusetts Association of Vocational Administrator, have filed a class-action lawsuit against the state of Massachusetts:

Even as vocational schools improve their scores and reduce failures to the single digits, advocates hope to change the law through the courts so that tech students are evaluated as much or more on their trade skills than on MCAS.

Lyons supports a class-action lawsuit that would allow schools to graduate vocational students who fail MCAS provided they demonstrate proficiency in their trade. Such students would be presented with a "certificate of occupational proficiency."

On the one hand, the vocational schools claim that their students are learning a trade, and thus deserve an exemption. To me, that claim carries more weight than their second reason, which is that the MCAS is "unfair" to votech schools because such schools have more "low-income and special-needs students." The lower-income students, at least, are the ones who are most in need of education and were the students in mind when NCLB was implemented. Wouldn't the MCAS serve at least as a reminder to vocational schools that, in this day and age, a student with a shoddy education in math and English is going to be at a disadvantage even if the student is good with his hands, or has learned a specific trade?

Some schools seem to have taken up the challenge and devote more time to academics (and to MCAS prep):

At Shawsheen, 300 students took the test last spring, 88 of whom are considered of special needs. Yet their proficiency in English language arts jumped since last year from 34 to 54 percent while math proficiency more than doubled from 14 to 35. Meanwhile, failures declined from 13 to 4 percent in English and from 41 to 18 in math...

Voke students across the state improved in English and math and saw failure rates drop.

Another form of motivation

The schedule for students at Shelbyville Junior High (IL):
Practice, practice, practice.
Repitition, repitition, repitition.
Testing, testing, testing.
Balloons, balloons, balloons.

Assessing scientific knowledge in Massachusetts

The first science, technology, and engineering portion of the MCAS was administered to students in Massachusetts this past year, and the public schools hope to use the exam as a guide to curriculum changes:

While other students were taking Massachusetts Comprehensive Assessment System tests in English and math last spring, fifth- and eighth-grade students took the first science, technology and engineering exam, which will be a permanent fixture on the test schedule from now on...

The tests help to prepare students to take science MCAS tests in 10th grade, which will become a requirement for graduation in 2006. Subject-specific tests in biology, chemistry, physics and technology/engineering will be given to 10th-graders as a pilot program in 2004 and 2005, but scores count in 2006.

Here are the disclosed 2003 science test forms for Grade 5 and Grade 8, if you're interested. The multiple-choice items seem pretty straightforward, although the subject material covered is so broad that the items seem to really jump around in terms of content, from ecosystems to the earth's crust to biology to auto mechanics. Really - item 12 on the eighth-grade exam asks which car part belongs to its control system, and if the answer is not, "Fuel tank," then I feel really stupid. [Update: Reader Chris tactfully points out that the control system probably involves the steering wheel, not the fuel tank. I tell you, my days with no caffeine just keep getting better and better.]

The open-ended items seem a bit tougher. Students must list the necessary items and steps for conducting experiments, describe animal and plant cell functions, and describe wind patterns. Straightforward, but certainly not something the students can BS or fudge answers to, I would think. The 32% number for the proficient/advanced group, while not stellar, is about what I would expect from today's public school students.

Weary in Carolina

South Carolina's students are confounding the state's DOE; the students improved substantially on math but dropped at every level on English, according to the most recent Palmetto Achievement Challenge Test (PACT) scores. Right now, almost 30% of SC's students do not score at even the "basic" level in English/language arts, a drop of 4 percentage points, whereas math scores increased by 6 percentage points. Educators blame test fatigue for the decline:

Something happened (in English/language arts) that we can’t explain,” said Sandra Lindsay, deputy state superintendent. “We’re checking everything we can think of.”

“By that time, they were losing focus,” said Mini Johnson, a fourth grade teacher at Lewis Greenview Elementary in Columbia.

That's possible. Educators have moved the English portion to the beginning of next year's exam, and if their theory is correct, the decline in scores will not continue.

Misunderstanding the meaning of test prep

Here's a well-intentioned, if slightly clueless, SAT article from Post-Standard (NY) writer Kate Perry. It seems she's convinced that the SAT is unfair because kids who pay for test prep might do better on the SAT than kids who don't:

It's no secret that the SATs are as much about knowledge of how the test works as about the questions on the exam. [By the way, this isn't true, but testing critics keep repeating it.]Students who are more acquainted with the test have an advantage over students who are unfamiliar with the strategy needed to crack it. This advantage comes at a price.

Students who can afford private tutoring sessions, books and high-tech calculators have a distinct edge over students who can't afford any of these things.

That this is stated as a serious criticism of the test amazes me. Let's take an analogous situation. You want to take a driver's license exam. You think you're naturally a bad driver, so you go to driving school, or you pay someone to teach you, or you just practice a lot. You end up passing the test, while a friend who did much less in the way of preparation failed to pass. You now have a driver's license, and she does not.

Is this proof that the driver's license exam is unfair? Don't be ridiculous. Test takers gain an advantage on virtually every exam under the sun (except perhaps for IQ tests) by preparing for that exam. The fact that the SAT, or any test, can be prepared for does not mean that the test is not a valid measure of acheivement or skills or knowledge. As this principal puts it, the key to better SAT scores is practice, practice, practice. That's also how you get to Carnegie Hall, should anybody ask.

What's more, this journalist buys wholesale the claim (made only by the test prep companies, not by the College Board) that she can expect a 200-point increase by taking a few practice exams - a few expensive practice exams, that is. If you refuse to grant her that assumption, and I do, then her entire argument that SAT point increases can be "purchased" falls apart.

What does it always come down to with these types of crusading journalists? Money. What this writer wants you to believe is that:

(a) it's inherently unfair that some kids grow up in richer families than others, when those kids have more options for academic practice open to them,

(b) it's possible to create any valid measure of academic achievement on which kids who are better prepared will not have an advantage, and,

(c) that no poor kid has a chance to do well on the SAT.

I believe (a) and (b) are bunkum, and it can be shown that (c) is pure poppycock. Every library in the US has practice SAT books, and these same books can be bought online, either from ETS or from booksellers, at a very reasonable price. Any kid who is discliplined enough to sit down with the tests, take many practice exams under the time limit, and ask a teacher or parent for help with the items they got wrong is going to get as much benefit from it as those rich kids whose parents can foot the bill for the test prep companies.

And that $28 test fee? Every testing company in the US allows for fee waivers from students who can't afford the bills. Some give partial waivers, some full; some high schools foot the bill as well.

Although I've written about this before, I still remain amazed that a journalist can get an article published despite the fact that they don't bother to do their research (re: cheap test prep and fee waivers), and despite the fact that they take the inflated claims made by test prep companies and restate them as criticisms of the tests themselves.

What I said to that other credulous reporter back in February still stands:

Last I checked, not only is there no solid research indicating that test prep courses work for all kids, but there's also no research showing that price is any indicator of the quality of a test prep course. Let me point out again that the test was designed as an equalizer and in fact works as one - but our society is unequal with regards to wealth, and always will be. A poor kid with great SATs will not be poor for long. A rich kid who can't pass the SATs will be dependent on familial influence and money to get into college. The kid who works the hardest and had the most ability will probably do the best. The test prep companies will charge whatever credulous customers...are willing to pay. And this is a criticism of the tests how? It isn't ETS creating that divide.

testing expert criticizes tests

One testing and assessment "expert" from UCLA has nothing but bad things to say about standardized tests, which leads me to doubt his expertise:

[Dr. W. James Popham], the emeritus professor of education at UCLA told teacher leaders at CTA's Summer Institute, you can't use standardized achievement tests to measure student performance. They're not designed to measure what's being taught or learned, but what knowledge or aptitude is being brought to the table. In fact, if a test item is answered correctly by too many students, it's taken off the test in the next revision. A teacher who does a good job and teaches students the material thoroughly will not be rewarded for the effort.

What? How does he conclude that no standardized test measures what has been taught or learned? How exactly do students gain knowledge if it isn't taught to them? Tests like the FCAT aren't measuring IQ-test-like skills such as digit memory; they're measuring how well kids were taught to read and understand passages. Teach a kid to read well and they'll pass the FCAT.

And the part about removing items that everyone does well on is bunk. Yes, a test question that every student answers correctly - i.e., a very easy item - might be taken off an exam, because it's not discriminating among students, but it won't necessarily be removed. It's not even likely to be removed if the exam is criterion-referenced; there are easy items on all standardized exams, and no school district is going to complain as the percentage of students passing the exit exam rises.

What's more, if easy items are removed, then students who are being taught more thoroughly are going to benefit from that, not be disadvantaged by it. It's the kids who aren't being taught well who are going to start flunking. No aspect of the standardized testing system disadvantages kids who learn more, or who learn better, except that they might be bored by dumbed-down exams.

What teachers want and need is "instructionally supportive tests that provide clear descriptions of what's tested; measure a modest number of curricular aims; and supply instructionally informative results." In other words, they need tests that give them information they can use to adjust their teaching and fill gaps in student knowledge in a timely fashion.

That, in and of itself, is not a bad idea, but I thought that was how teachers were supposed to be testing in the classroom. If teachers never give any exams except those mandated by the government, of course they're not getting feedback in a timely fashion. If teachers need feedback, they give their own exams. The large-scale exams provide a snapshot of how large groups of students are doing at one time, and while this can provide some feedback, it was never intended to provide all that a teacher needed to be effective. It does, however, indicate how schools are doing overall, which is what NCLB-related exams were meant to do.

Teachers are in a predicament, said Popham. They can't complain about the tests too loudly. "Attempts to have the [test] replaced by instructionally supportive assessments will be seen as a flight from accountability."

Actually, I've read about few teachers who claim that the accountability standards should remain, but with different tests in place of the standardized tests. The teachers who are accused of fleeing from accountability are the ones who claim that their school's F grade is just fine, or who claim that they simply can't be expected to actually teach their young charges to read, given the race, or background, or SES of said students. Those teachers are attempting to avoid any objective measure of their teaching skills, and the anti-accountability label is apropo in those cases.

[Teachers] need to be able to explain the likely impact of unsound tests, why inappropriate tests give schools inappropriate labels, how tests may measure what students bring to school rather than what they learn, and how to judge what schools are doing the right way. "We have to educate parents and pertinent policy-makers about the evaluative inappropriateness of standardized achievement tests. And we have to collect convincing evidence signifying that students are learning worthwhile things."

Yes, they do need to understand which tests are unsound, but they also need to understand that not all of them are. I fail to see any information here which supports the argument that standardized tests are, in and of themselves, unsound. This sounds suspiciously like the same "holistic/portfolio assessment" argument in new clothes, which, given that Dr. Popham's arguments are allegedly about keeping standardized tests from eating up class time, is especially disingenuous. How would many little narrow performance assessments not eat up an equally-large amount of time, if not more?

And how can any test be given if the concern is all about students who are "poor test-takers?" The focus on that suggests that no testing can be done at all, and any teacher who argues for that is indeed in flight from accountability. And if Dr. Popham is convinced that standardized tests measure only what kids "bring to school", isn't he saying that teaching can have no impact whatsoever on how well a child can read or do math? That sounds much more like an indictment of the teaching profession than of the testing profession to me.

Tilting the SAT playing field

There's a fascinating article over at Education Next entitled, "Disabling the SAT." It's long, but well worth your time.

Those of you who are regular readers know that I have long been critical of some issues surrounding accommodated testing, most especially the decisions by testing companies to discontinue the flagging of tests that are given under extended time limits. Those of you who wish to review my comments on this can click here, and, especially, here.

Done? Good. Back to the EN article. Attorney Miriam Kurtzig Freedman has answered my prayers with a long and thoughtful look at the recent ETS/College Board decisions surrounding accommodated testing, and the lack of public controversy surrounding the concessions on the part of testing companies is as surprising to her as it was to me:

In 1999, after taking the Graduate Management Admission Test (GMAT), the standardized exam required of applicants to business schools, Mark Breimhorst sued the test’s maker, the powerful Educational Testing Service (ETS). Breimhorst was born without hands and thus had been given more time to complete the admissions exam. His lawsuit contested ETS’s practice of informing schools when students take one of its tests under specialized conditions, effectively placing an asterisk or, in testing parlance, a “flag” next to their scores. For unexplained reasons, instead of weathering a trial, ETS settled the case and agreed to stop flagging GMAT scores.

Once the GMAT was no longer flagged, disability rights activists went after the SAT, claiming that these flags "stigmatize" disabled test takers. As I posted previously, a flag is a "stigma" only if we are willing to concede that admissions officers choose to discriminate against handicapped applicants, which is hard to believe in this day and age. Removing the flag, on the other hand, is a surreal denial that the test was given under non-standard conditions in the first place. At the heart of the matter is whether choosing to interpret a non-standard test differently from a standard test is "stigmatizing," or sound psychometric practice.

Accommodations like extended time, [disability rights advocates] believe, are necessary to equalize the testing experience for disabled and nondisabled students and thus make the scores of disabled students [on the SAT] more valid.

The problem is, those scores haven't actually been validated, not in a statistical sense. Equalizing the testing situation in this sense doesn't automatically translate into equalization of the predictive validity of the two types of scores. There's a great deal of controversy over whether extending testing time actually "levels the playing field" for disabled students, as well as whether accommodations can benefit non-disabled students who obtain fraudulent diagnoses. If the former is false, there's no reason to offer accommodations; if the latter is true, the potential for fraud is great. Neither of these, though, have any bearing on whether admissions officers should know whether a test was non-standard.

Should a test be flagged if it's given in Braille for a visually-disabled test taker? Not necessarily, because a credible argument can be made that the same skills are being measured. Reading comprehension is reading comprehension, whether you read with your eyes or your fingers. But because admissions tests are almost always speeded, changing the time limit of the tests will almost certainly change the nature of what's being tested, so extended-time tests should be flagged. The only question that needs to be asked when considering the removal of these flags is, "Do non-standard tests predict college GPA as well as standard tests?" If they do, there's no reason to flag. If they don't, it's fraudulent not to flag, all the talk about "stigma" notwithstanding.

As Ms. Freeman notes,

"The College Board website tells aspiring matriculants, “Your scores show colleges how ready you are to handle the work at their institutions and how your verbal and math skills compare with those of other applicants.”

Therefore, no scores should be used unless they have been validated in this respect. Regardless, the College Board appointed an expert panel to discuss the removal of the flags, and ultimately they did remove them (with their main competitor, ACT, following suit only weeks later).

How many test-takers does this affect? Only 2% of SAT-takers request special accommodations - but almost all of them request additional time, and for some, such as those who claim learning disabilities, this is the only accommodation they request. As for learning disabilities (LDs), these diagnoses have increased 300% since 1976 - and while the percentage of SAT-takers has increased only 18% since 1987, the percent of those requesting accommodations has increased by more than 300%. (Don't miss the accompanying graphs, by the way, which suggest that this increase in the requests for accommodations have been accompanied by an increase in test scores for disabled test-takers.)

When the number of people in any situation who claim to be disabled increases at a rate this much faster than the increase in the general population, either something is happening to create disabilities, or a new way of diagnosing disabilities is present - or some people are not being honest about their disability status. Removing the "stigma" of the flag for extra time (which is the accommodation that LD test-takers will virtually always request) will do nothing to stem any tide of fraudulent claims.

Ms. Freedman goes one step further than (correctly) criticizing the College Board's puzzling decision to stop flagging on the SAT. She suggest three alternative routes that could have been taken:

Untimed SATs for all. If, as the College Board asserts, time doesn’t affect validity, administer the SAT untimed for everyone! As she notes, this would still change the nature of the test - but at least it would change it for everyone, so that everyone's scores could be lumped together for the validity studies. However, despite the claims of the disability advocates that they simply want disabled test-takers to be on equal footing with everyone else, I have the feeling that few of these advocates would go for this truly leveling option, as opposed to one that gives preferential treatment to disabled test-takers.

Let the students decide. If time does affect validity and standardized norms, as we have been led to believe since the precursor of the SAT began in 1926, then the Board can avoid the allegation of discrimination by allowing all students to choose whether they want extended time. No. Given the ease with which some students have obtained fake LD diagnoses, this is essentially what ETS and the College Board are already doing. Also, this doesn't remove the problems of the necessity of flagging and the differential test validity. Besides, I'm cynical enough to believe that some test-taker will be willing to sue on the grounds that being forced to choose is in and of itself stigmatizing, if the flags are still in place.

Defend the SAT. The College Board could void the settlement. If actually sued, it could defend the SAT in court...a court would most likely defer to educational experts, uphold standards supported by evidence of the SAT’s validity, reliability, and technical underpinnings, and find flagging not to be unlawful discrimination. This is, not surprisingly, the option that I believe College Board should choose, not least because this is the only option that conforms to the Standards for Educational and Psychological Testing set forth by American Educational Research Association (AERA), the American Psychological Association (APA), and the National Council on Measurement in Education (NCME):

When there is credible evidence of score comparability across regular and modified administrations, no flag should be attached to the score. When such evidence is lacking, specific information about the nature of the modification should be provided, if permitted by law, to assist test users properly to interpret and act on test scores.

The proof of "credible evidence" should have been the lynchpin of the expert panel's decision to remove the SAT flags, but Ms. Freedman's summary of their conclusions makes it devastatingly clear that this decisions was not guided by research, nor by sound psychometric theory:

The bottom line is that the panel majority had no research directly comparing changes in the performance of nondisabled and disabled students when both are given extended time. Moreover, the existing research, given the limitations outlined above, hardly establishes that many nondisabled students would not benefit from having extra time. This is not to criticize the research, since the researchers themselves acknowledge these limitations. The problem is with the panel majority’s drawing of firm conclusions based on inconclusive evidence...

The two psychometricians on the panel certainly recognized this. In a joint statement accompanying their individual reports to the panel, when they asked themselves the key question—whether scores from the SAT taken with and without extended time are comparable—one answered “no” while the other said “not sure.”

What's more, the panel quibbled over the definition of "credible evidence," and focused on the fact that the research did not conclusively show that the two forms of testing were different, when in fact the decision to remove the flags should have been made only if the evidence conclusively showed that there was no difference in predictive validity. I was previously unaware of the exact breakdown of the vote, but this outline makes it quite clear that the decision was primarily based on the urging of the three panelists "with experience and training in the special-education and learning-disabilities arena." Funny, I don't remember seeing anything about that in the Standards for Testing guidelines.

The conclusion?

With the decision to end flagging, most students will take the test in three hours, some in four and a half hours, others in five hours (for a shorter version of the test), without any reporting of these differences. Does that pass the test of common sense?

No. As noted in the article, speed counts in the real world, admissions officers would prefer to have the flags for making decisions, and the potential for disability-claim abuse will only increase. The playing field is now more unequal; students who receive accommodated tests, even if not truly disabled, will most likely have an advantage over students who do not. It's hard not to conclude that this is what the disability rights advocates and people "with training in the learning-disabilities" have wanted all along.

Grudge match: Students vs math teachers

Now, I'm all for giving students non-monetary incentives to improve their academic performance. However, I find this a little....skeevy:

In an effort to encourage ninth-graders at East Ascension High School to take a standardized test seriously last year, a poster was circulated at the school promising students they would have a chance to dunk Algebra I teacher Judy Braud. The advertising gimmick worked as 156 sophomores made the grade...

The students were rewarded with a day in the sun for earning a higher score on the ninth-grade IOWA than they did in the seventh grade. "All they had to do was improve by one point and they were eligible for the fun day," she explained.

I understand that the IOWA is used for school ranking and not student ranking, and therefore students might not take it seriously. But still, giving credit for a one-point increase in scores? That makes it obvious that the school wasn't comfortable with rewarding only those students who made meaningful improvement, but was willing to reward those who basically stayed the same (the one-point increase is most likely well within the standard error of measurement).

Plus, if the kids did better on the test, why make one of the rewards a kind of "punishment" for their teachers? Yes, I know dunking is in good fun - but isn't it nice to imagine a situation in which teachers get dunked if their students do worse on the exam? Doesn't it make more sense for a student to want to dunk a teacher who shortchanged them, rather than one who helped them improve their scores?

Grudge match: ACT vs SAT

The Associated Press is adding 2+2 and coming up with 3 when they compare the recent releases of nationwide ACT and SAT scores. The SAT says math scores are at a 35-year high - but the folks at ACT say that this year's incoming college freshmen are unprepared for college-level math courses. Are these two conclusions necessarily contradictory?

The ACT scores for the high school class of 2003 were identical in math and science to the year before - 20.6 and 20.8, respectively, on a 36-point scale. In the last five years, math and science scores have dropped slightly on the test, taken by nearly 1.2 million of last spring's high school graduates. Researchers for the ACT analyzed this year's results and concluded that just 26 percent of test-takers were ready to handle college coursework in science and 40 percent in math.

Meanwhile, the SAT math scores were the best since at least 1967: 519 on a scale with a top score of 800. Since the 1999 exam, math scores are up eight points.

So scores are rising on the SAT - but not on the ACT. Has the SAT been getting easier? Are there college performance indicators that indicate which test is more accurate? Some say yes:

Some educators say the number of students enrolled in remedial math and science courses at four-year schools support the ACT's conclusions. Michael Kirst, a Stanford University education professor, said the ACT's position is compatible with a study he co-authored earlier this year. It also found that many freshmen are not prepared for college math and science, despite gains in achievement scores.

"To have scores higher than 35 years ago and to be testing a larger and more diverse student body than was tested 35 years ago is pretty darn impressive - whether they're ready for college or not," Porter said.

Eh, I don't know about that. Yes, it's impressive that SAT scores have gone up, but SAT scores are supposed to predict readiness for college. SAT scores are supposed to predict first-year grades in college. If high SAT scores aren't predicting that, or if students with high SAT math scores are in remedial math classes, then something is wrong. I think it does matter if students are "ready for college or not."

Of course, the SAT and ACT populations may be different. The ACT packs in the Midwestern crowd, while the East and West Coasters are more likely to take the SAT. Ivy League colleges are more likely to require the SAT as well.

The ACT decision that today's high school seniors was based in part on questionnaires that showed fewer than half had four years of math classes. But how many of those with only Algebra I, II, and Geometry under their belts planned to attend college? What's more, the College Board reports that high school precalc course enrollment is up 12% from 1993.

So what's going on? I disagree with the cynical view that this is all due to marketing. One explanation could be that the SAT test-takers now include enough stellar students to move the mean upwards, while the ACT does not. The ACTs estimation of student readiness could be incorrect. Or, it could be the case that even though students are testing better on standardized math tests, those scores are the result of better preparation in the lower math courses, and not necessarily an indication of whether the students are ready to do college-level math work. I don't have the answer, but I'm not impressed by the AP's willingness to let the question be cynically and abrubtly dispatched with claims of marketability and test promotion.

Teaching vs testing in rural north carolina

The Smoky Mountain (NC) News has a thoughtful article on the struggles of teachers in rural mountain schools to comply with NCLB, and their concerns about the "standardized testing culture":

“Here we go again,” Tammy Bates thought as she passed out the second standardized writing test in one week to her 10th-grade English students at Tuscola High School...

In the back of Bate’s mind, she wondered what would go wrong with the test this time. A week earlier, the state told teachers to administer a field test and that a pilot test would come later. Later happened to be the following week, affording no time to discuss the pilot test with students and work through problem areas. The previous year the writing topic was “diversity,” something there isn’t a lot of at rural mountain schools. The topic was deemed too subjective and the marks thrown out.

The article describes the rushed time schedules, inappropriate test content, and the stack of new national standardized tests that have been piled onto a school system that already had a meaningful local standardized test system in place. What's more, the NCLB-mandated division of students into ever-more-precise racial and educational groups doesn't work the same way in these rural schools:

Western North Carolina lacks some of the profile groups found in larger cities. Profile groups that are present in rural mountain schools often exist in smaller numbers. This works both for and against the region in meeting the No Child Left Behind standards.

But at a rural school, there might be only 40 students in a profile group. If 15 fail their reading test, they don’t make the 76 percent mark, and the entire school gets a failing grade.

All this additional bean-counting is frustrating the teachers, because it isn't telling them anything new:

For all the turmoil and aggravation, the No Child Left Behind standards tell educators and politicians very little that they didn’t already know. Low-income children, minorities and disabled students do not perform as well on tests as white, non-low income students, according to state test results on ncschoolreportcards.org.

“Schools have been dissecting this data for years. In North Carolina, people can be frustrated and rightfully so because we already had a testing regime in place,” [Director of research John] Poteat said.

Figuring out the ACT in Colorado

Want to know more about interpreting ACT scores - and about how well (or poorly) a school is doing? Ari Armstrong, posting for the Colorado Freedom Report, has a discussion about the performance of Colorado's schools on the ACT. He graciously allowed me to make some comments about the interpretability of ACT scores and the true distance (in terms of academic achievement) between Colorado's best-performing schools, and its worst. Go, read, enjoy.

Too much of a good thing in South carolina

South Carolina's young'uns sound plumb worn out by the battery of tests they face each year - at least one in every grade, including kindergarten:

The state requires five standardized tests — the Palmetto Achievement Challenge Test, the South Carolina Readiness Assessment, the BSAP Exit Exam, the High School Assessment Program and end-of-course tests for four high school-level classes. Two other tests — the TerraNova and the National Assessment of Educational Progress — are given to samples of students. Greenville County requires two more — the Iowa Test of Basic Skills and the Iowa Test of Algebra Aptitude.

That means some eighth-graders could take as many as six tests this year.

That's too many. The folks in SC are sounding pretty civilized as they call for reform, though; they don't have a problem with testing, but they don't see the need for so much of it, especially when there's overlap. And one education professor at USC has some practical solutions in mind:

...there should be a way to streamline testing.

"I think we need to test to make sure kids are learning what they should be learning, but I don't think we have to test all the kids all the time," Anderson said.

Making the grade

A Devoted Reader alerted me earlier today to "Making the Grade," the Discovery Times Channel's documentary on testing and the NCLB Act. I haven't had the chance to watch it - but dig this review that has already appeared in the Palm Beach Post:

...the Discovery Times Channel's special Making the Grade (8 p.m. today) [is] a searing look at the high-stakes world of standardized-test scoring. "We regulate dog food and we don't regulate these companies," an industry watchdog says during the one-hour special.

"Searing?" My, I never knew my field was that exciting. I find it hard to believe that even the most dogged of investigative reporters would have used that word after following a few psychometricians around for a day...

The directors point their camera at a world in which four companies -- CTB McGraw-Hill, Pearson NCS, Riverside Publishing and Harcourt Educational Measurement -- control 95 percent of the nation's test preparation and scoring. It's a "cartel," says one state's school superintendant, and it gobbles the largest slice of a $730 million pie.

Four independent companies, and it's a "cartel?" The last I checked, these four companies were in competition for their products, which would rule out their working together to control prices and supply. What's more, it's not the large companies themselves that prevent the smaller companies from gaining a bigger slice of the pie, but the public's demand for speed and accuracy. After all, if a giant like NCS Pearson can make a big mistake, as they did in Minnesota, why would a school district trust an even smaller company, which would presumably have fewer QC controls in place?

To comply with a 2001 federal law, most states, including Florida, instituted massive end-of-year tests. Spring might have most people here thinking swimsuit shopping, but for test companies, high season means a deluge of pencil-flecked answer sheets. For their efforts, Making the Grade gives them a fat, red F, claiming their errors have affected 1.5 million students and rankings or bonuses for 4,000 schools.

Thus we are treated to a rehash of the NCS Pearson errors (which were substantial) - but does the documentary note, as the NYTimes did, that the demands of the school districts help to create a situation in which there wasn't enough time for adequate QC? Does it mention that errors would be expected in any enterprise that has been geared up to the extent that standardized testing has been over the past few years? The question is not, "Were mistakes made?" but, "Were more mistakes made than would be expected, given the limited time most companies were given?" and "Can we live with the results?" I'd say the testing industry has made fewer errors than you'd expect, given that some tests were ordered only three months before they were needed - but the results of errors have indeed been high-stakes, and there's the rub. Can accountability wait until the testing process is "perfected"?

Refusal of the four largest test companies to participate leaves the viewer wanting more, but the documentary offers in their place an illuminating behind-the-scenes peek at a fledgling test company that welcomed the publicity. That company's execs offer an earnest appraisal of the pressures they face from increased testing.

I'm not surprised at the refusal of these four companies to open their doors to scrutiny, thanks to the fact that testing reporters are rarely unbiased - but kudos to the fledgling company that spoke with reporters.

The Discovery Times Channel website contains a primer on testing that isn't bad, though there are plenty more subjects it could have covered, and it's surprising that the site omits the fact that computer scoring of essays is one of the biggest areas in educational testing research right now. Even the test prep companies know that on some high-stakes exams, computerized essay scoring is already in place.

Anyway, as I said, I haven't watched the thing yet, and might have to take some blood-pressure medicine before I do so, in case I feel the need to put my fist through the TV screen. Hopefully, it won't come to that.

Buried under the mess on my desk

Hey, everyone. Hope you had a good Labor Day weekend! Mine was not as relaxing as I'd hoped, and now the work is piled up on my desk. There's a whole bunch of little postings and a couple big ones I hope to make today, but we'll have to wait and see if I get to them...

The NAACP in Florida: Update

Yesterday, I noted that the NAACP is opposing the use of the FCAT in Florida, in spite of the fact that FCAT performance gives them the evidence they need to urge schools to better educate their minority youth. John Rosenberg of Discriminations comments, and he's noticed an interesting contradiction between that and NAACP action in California:

In Florida the NAACP does not want the state to collect information that would identify poorly performing schools. By contrast, in California, as one of their primary arguments against Prop. 54, the Racial Privacy Initiative, the NAACP and its allies argue that the state must continue to collect racial information in order to monitor compliance with civil rights laws.

He also provides a link to the NAACP brief, which claims that Florida systematically preserves segregated schools, but their evidence is given only as the policies that are in place for standardized testing, AP course placement, graduation requirements, and the like. Unless I'm completely off here, those policies are completely color-blind in Florida's schools. The standardized tests, in particular, are color-blind, but that's not what the NAACP wants. They want some method that produces equality of outcome, not opportunity, and they, like many test critics, believe that banning tests changes the fact that a score gap exists.

Den Beste's theories that I mentioned earlier today provide an explanation that has everything to do with economics, and nothing to do with a racist educational system. The fact that Florida's public schools are highly segregated are more likely due to "white flight" and suburban retreat than to any systematic policy on the part of the Floridia DOE.

If Florida is in fact segregating and mistreating minority youth, then one would think the NAACP would want the FCAT should remain in place as objective evidence of that. Page 25 of the brief deals with the FCAT. Although the writers of the brief do correctly label the results as differential impact, and not bias, they're still wrong in assuming that evidence of differential impact is a de facto criticism of the test, or that it's evidence the test should be removed.

More ruckus in Florida

The NAACP has gotten into the act:

The NAACP filed a federal complaint seeking to stop use of the Florida Comprehensive Assessment Test, or FCAT, to determine graduation and student retention until the achievement gap between minority and white students is eliminated.

Wow. That's quite a requirement. Equality of opportunity is no longer sufficient; now minority and white students must have completely equal distributions of ability for the test to be considered "fair" by the NAACP.

...Education Secretary Jim Horne said that the gap between white and minority achievement is closing. He had not seen the complaint, but said the National Association for the Advancement of Colored People has a history of agitation.

The NAACP claims that Florida is violating its constitutional duty to provide a quality education to all students, saying schools that are mostly minority are not receiving the same resources and have inferior facilities.

It's not that they don't have a point with this. It's just that, if you remove objective measures such as the FCAT that show minority youngsters are not gaining the skills they need, then you lose critical evidence that some schools are shortchanging their students and do need to change. Why does the NAACP want to get rid of evidence that minority students aren't learning what they need to learn?

Note to NAACP: Quit attacking the test, and go after schools that have the lowest FCAT performances. Demand to know why these schools can't help their kids learn. If the teachers or administrators claim that they cannot overcome the home lives and backgrounds of their students, then you have the cause of the score gap right there - teachers who have abdicated their responsibility to teach, and schools that have abdicated their responsibility to help kids overcome a poor home environment. Fix that, and you'll see the score gap close. Eliminate the FCAT, and nothing will happen to improve education for minority youth.

Misunderstanding bias at the NYTimes

Devoted Reader Nick sends along an article about the protest against teacher certification exams in New York City. Not surprisingly, the protestors are claiming that the test is biased against those who failed; also not surprisingly, the NYT reporter who covered this story fails, along with the protestors, to grasp the distinction between bias and impact.

The protesters urged the city to reconsider its firing of 10,000 uncertified teachers over the last five years, saying the test was culturally biased against blacks and Hispanics.

Marc Pessin, a co-chairman of a teachers' group called the Progressive Action Caucus, which organized the demonstration, said the passing rate among blacks and Hispanics was about 40 percentage points lower than that among whites.

If the passing rate is different for blacks, whites, and Hispanics, then this means the distribution of scores are different. The black and Hispanic distributions are most likely shifted lower on the score scale continuum, so that a smaller percentage of those members fall above the passing cutscore than in the white distribution.

This is NOT evidence of bias; indeed, bias can exist where group mean differences do not. No reporter seems to understand this, and no testing opponents want to believe it. The differential passing rate means only that the groups differ on the ability scale. There are many explanations for why that would be; wholesale "cultural bias" is one of the least likely, and, as I'll explain in a moment, one of the most distasteful explanations.

There is not any firm evidence of test bias here, but because black and Hispanic teachers are passing at a lower rate, using the test for certification purposes does have a greater negative impact on those test takers. Impact is, in itself, a neutral term, and not a criticism of a test. Here's why.

Example: Group A and Group B take the driver's licensing exam. 90% of Group A passes; 15% of Group B passes. Is the test biased against Group B? No, but the test negatively impacts Group B. Suppose you then discover that Group A has had driving lessons, while the members of Group B have never sat behind the wheel of a car. Is it a bad thing, then, that the exam has a differential impact on the two groups? Not at all; in fact, the results are evidence of the construct validity of the exam, because the results show that the exam prevents people who don't know how to drive from obtaining licenses.

The analogy holds here. Black and Hispanic teachers do not pass the test at the same rate. It is not racist to say that, for some reason, these teachers are less likely to have the skills necessary to pass the exam. Perhaps they were not taught them; perhaps they did not believe they would ever be tested on them.

It is racist to claim, as the representative of the "Progressive Action Caucus" quoted above does, that these teachers are "culturally" unable to pass this exam based solely on their skin color. Saying that black and Hispanic teachers are not capable of answering multiple choice items for reasons that have nothing to do with their skills, and everything to do with their race, is hardly a "progressive" idea. In fact, the Klan would likely agree with the "Progressive Action Caucus" on that point.

(More on bias and impact here, here, and here.)

What does the test look like? See for yourself. Here's the test framework for the Liberal Arts and Sciences section, and sample test questions can be found here. The questions ask about topics such as equilibrium, linear relationships, a basic understanding of experimental design, and understanding mathematical concepts that have been graphed. The English prep guide is here; sample multiple-choice items begin on page 24. The reading load is substantial, but only people who think teachers shouldn't know how to read English very well would think that's a problematic aspect of the test:

Jose Aguasvivas, a former math and Spanish teacher in the bilingual program at Roosevelt High School in the Bronx, said he was typical of teachers who had failed the test because English was not their first language. "I even have a master's degree in bilingual elementary education," he said. "But the test is very confusing. If the test is in Spanish, then I pass it no problem."

Are you teaching in a Spanish-speaking country? Are you supposed to be instructing students only in Spanish, and never in English? If not, then this justification isn't satisfactory.

Another teacher laments the fact that she failed the test after teaching for 19 years and has had to go on welfare as a result. One could conclude the test was unfair; one could also conclude that this teacher was not very skilled to begin with, as evidenced by her failure to pass the exam, and her failure to find another job.

So, how difficult is this exam? This letter claims that most teachers pass on the first attempt. This page gives the passing scores, but not the total scores on each section. Teachers can retake an unlimited number of times, but must pass each section on one attempt to receive certification (they cannot combine passing sections from multiple attempts). This FAQ from a test prep site claims that teachers need answer correctly only 50-55% of the multiple-choice items in order to pass each section of the exam (they must pass the essay sections as well).

I dispute the claim that the test is biased, but do we know for sure that the exam is useful for identifying good teachers? Not necessarily. At least one study failed to find a link between teacher certification and student test scores - but DID find that the strongest effect on student performance was due to "teacher verbal and cognitive ability," which is certainly something that can be incorporated into a certification exam.

How do you think the "Progressive Action Caucus" would react if NYC suggested adding an IQ test to the certification procedure, or incorporating SAT scores? The fireworks would be fun to watch.

Some positive testing news

The class of 2003's SAT Math and Verbal scores have increased dramatically over the scores of previous classes of students:

The College Board, which owns the nation's most popular college entrance exam, said Tuesday that this year's high school graduates had an average cumulative score of 1,026 points on the SAT, up six points from 2002. Both the average math (519) and verbal (507) scores were up three points from last year.

That may not sound like much, but we're talking mean shifts here. Means don't shift that easily, especially not with sample sizes in the millions. What's more, minority test takers now make up 36% of the sample, as opposed to 30% 10 years ago, which contradicts the argument that the presence of minorities was guaranteed to explain the downward drift in mean scores.

This year's average math scores are the highest the College Board could document since 1967. Scores prior to 1995 were recalculated to reflect changes made that year so that the numbers would be comparable to more recent scores. The board was unable to provide comparable scores prior to 1967. The SAT was first given in 1926.

The College Board said the higher scores were due to increased enrollment in advanced math and science courses such as physics, precalculus, calculus and chemistry.

Right on. Of course, there's some blather about how this is the result of not teaching kids "pure calculation" methods, as though a focus on the basics of calculation was the reason for the Math SAT decline in the first place. The article also notes that young women's Math scores increased "notably" - but racial test score gaps still persist (as will, presumably, the claim that the test is culturally or racially biased).

Let's look at some numbers:

* The means have indeed climbed up - but they're just above where they were in 1972. Too many years of "progressive" and ineffective teaching techniques have taken their toll, but now that accountability is in the picture, we're seeing some positive results.

* Interestingly, almost half of the SAT-takers report having a B average, and 72% are in the bottom 90% of their class.

* The score gap is on page 6. Self-reported White students have the highest Verbal average, while Asians have the highest Math. In fact, the average for Asian females on Math is higher than for White males.

* Note that the "Other" and "No Response" categories aren't doing too shabbily. While the "Other" is only 4% of the sample, the "No Response" group contains over 355,000 kids, which would be almost a quarter of the sample. The "No Response" group has the second-highest overall Verbal mean and the third-highest Math mean. Evidence that kids who refuse to play the "group identity" game are smart? Or just sick and tired of being asked about race?

* More trivia - 2% of SATs are taken under special accommodations. 13% of SAT-takers come from families with incomes less than $20K/year, and 38% of them come from families where the parents did not attend (or complete) college.

* Oh, and there's always the fun part of comparing average combined scores for varying majors. Kids who want to go to college and major in biological sciences have an average of 1096; future philosophers or theologians, 1106; the little engineers that could, 1099. The bottom four averages? Technical and vocational, home economics, education, and agriculture.

* There's an equal or higher percentage of men than women in every score category from 550 on up.

* Size of senior class is almost perfectly correlated with SAT mean scores, except for that group who go to really tiny schools. Students who come from schools where the senior class was composed of a mind-boggling 1000 students or more had the highest mean scores; students from schools where classes were 100-249, the lowest. And check out those mean scores for Independent and Religiously Affiliated schools, vs the public schools.

Freaking out in Florida

The Smarter Cop lays some tough love on a group of FCAT-bashing Floridians who are so over the top with touchy-feely Concern For The Children that it almost seems like satire:

------------------------------
From the article: "Accountability can be accomplished without fear and intimidation," said Brenda Katz, one of the RATPACK organizers. "When I hear about gifted students getting sick and not being able to sleep the night before the test, something is wrong. And we need to do something about it."

The Smarter Cop: Excuse me, did you say fear and intimidation? Is your child ever going to encounter a challenge in his life? Maybe not while he's in your house, he's not, but once he gets out into that cruel world he's going to encounter tests more substantial and threatening than the FCAT. These students are being taught that challenges and tests are unfair; their fear of failing is coddled and has been imbedded so deeply into their persona that it affects them physically.

The article: "Our schools are getting funded by how well my child performs on a test," Janisch said. "In my book, that's prostituting my child. I can't stand by for that."

TSC: You're darn right, missy. The schools you want to educate your child should be competent and trained enough to teach your child what he/she needs to know to enter the real world. Those tedious, sometimes boring English, math, and science classes actually mean something outside of school. All the FCAT does is verify that your child has a grasp of what's supposed to be taught - the foundations of the grade level they're at...

The article: "We're pitting school against school and teacher against teacher," Corbett said. "What I saw tonight was what I needed to see."

TSC: There's absolutely nothing wrong with competition! It provides motivation, pride, and enthusiasm in the students and teachers - or at least it should, before a group of whiney parents decided it was 'unfair' to have to compete because you might... uh... lose. It's not a race, though; unlike a typical competition, there doesn't have to be a last place if everyone performs at an 'A' level...
------------------------------

You know, I was just reading today an old op-ed about George Orwell's magnificent essay, "Politics and the English Language." Mr. Goldberg writes:

Orwell's essay could have been written today, and if you haven't read it, you should. Indeed, some of his observations are flatly depressing because they reveal how bad things have been for so long. For example, Orwell writes: "The word Fascism has now no meaning except in so far as it signifies 'something not desirable.'" Anybody who's listened to some college kid — or professor — denounce some new on-campus parking policy or a change in financial aid as "fascist," knows exactly what Orwell's talking about.

Yes, just as I know what he's talking about when mothers in Florida claim that "fear and intimidation" are involved in requiring kids to sit still for a while and answer multiple-choice items on a test, or that using the educational achievement of students to assess their schools is "prostitution."

And I feel sorry for this kid, although not for the reasons the mother would wish:

"My daughter is an honors student and has a 4.0 grade point average," Lilly said. "The night before the test, we had to take my younger son to the emergency room. We had to wake my daughter up in the middle of the night and take her to a friend's house.

"She missed passing the reading portion of the FCAT by two points because she was concerned about her brother. She was embarrassed and devastated, and now she suffers from test anxiety."

Well, Mom, that was the point at which you should have told her, "Hey, kid, these things happen. Sometimes life throws you a curveball. You were worried about your brother, and that's good. That was your priority, and it wasn't your fault that you had to miss some sleep. You'll get the test next time around."

However, if the mom was emphasizing the missing-by-two-points thing and carrying on about what a horrible burden this was to place on her child, well, that's not gonna help with test anxiety. The kid's an honors student and has a 4.0 average; have some faith in her abilities to overcome obstacles. She'll get the test next time around, but only if she's motivated, not scared out of her wits and convinced that she's being "prostituted."

I understand the concerns of these parents, but they're not helping anyone with these overprotective exaggerations; not their cause, and definitely not their kids.

Mind-boggling scores in Philadelphia

Students in the Philadelphia area (which includes Philadelphia County and four surrounding counties) have made some impressive improvements in reading and math this year:

Scores of schools across the five-county Philadelphia area made significant strides in the number of students who are proficient in math and reading, an Inquirer analysis shows.

And schools lagging in performance last year posted some of the best gains. In fifth-grade math, for example, 117 of 479 schools in the region that gave the tests improved proficiency scores by 10 percentage points or more. In reading, 94 schools improved proficiency by 10 points or more.

Some improvements are so good they're hard to believe:

The Showalter Junior Academy, a middle school run by Edison Schools Inc. in the troubled Chester-Upland district, made remarkable progress in both reading and math.

Seventy-one percent of the school's eighth graders reached proficiency in math, up from 11 percent the year before. On reading tests, 82 percent reached proficiency, compared with 57 percent in 2002.

This is Edison's first year running schools in Philadelphia, and while not all Edison schools showed this kind of improvement, at least for this one school, something is being done right.

Admiration for these results must be tempered by concern for how far some students in Philly, and elsewhere in Pennsylvania, have to go:

Citywide, students performing at the proficient level or above in math went from 19.6 percent in 2002 to 21.6 percent in 2003. In reading, students reaching proficiency increased from 23.9 percent to 27.5 percent, according to school officials.

The picture statewide was less upbeat. Just more than half the incoming class of 12th graders - 67,800 students out of a total 133,200 - failed to reach proficiency in math, while about 54,300 students fell short on the reading test.

The results suggest that only a little more than one-fifth of Philly's students are proficient in reading or math. That's insane. How did things get that bad in the first place? The students who are really struggling - performing below grade level - are now required to get additional help:

...starting in September, participation in after-school and summer-school programs will no longer be voluntary for struggling students. Those in third, eighth or 12th grade will be required to attend the programs to be promoted to the next grade.

Amy Guerin, a school district spokeswoman, said the district would not know how many students were retained at the end of the 2002-03 school year until it had received the results from the district's summer-school program...

Given those test scores, I'd say it's going to be a large number, and I'll report that on here when the figures are released. Perhaps a massive number of students forced to repeat grades will be yet another wakeup call to Philadelphia parents and educators.

Watching the gap close in NC

Moore County (NC) school superintendent Pat Russo believes the black-white test score gap is closing in his district, and his remarks to parents who are also NAACP members sound like the voice of wearied reason:

Russo acknowledged that the No Child Left Behind program, North Carolina’s ABCs of Public Education, and other yardsticks for comparing schools are based on the progress that students make on standardized tests such as the Scholastic Assessment Test and end-of-grade tests. But he sees nothing wrong with the often-criticized technique of “teaching to the test,” he said in response to a question from the audience.

“I’ve had many teachers — in other places; I don’t think I’ll hear it here — tell me, ‘I just didn’t have time to get through the curriculum. Why did 77 percent of my students fall below grade level on my subject?’” Russo pounded his palm against the side of his head, the equivalent of responding “Duh” to these teachers.

His head must be pretty sore by now. His belief in the closing of the gap is based on reports from school principals in his district, but the SAT data that would confirm his belief will not be released until next Tuesday. On that measure, at least, his schools are ahead of the pack:

The average score statewide on SAT tests is 998, he said. The average for the Moore County Schools is 1044. A perfect score is 1600.

The local school district scores will probably be about the same this year, Russo said. The Moore County Schools have already climbed from an average score of 954 in the 1998-1999 school year, when it ranked 70th among the 117 school districts in the state. It now ranks seventh, and will rank “somewhere between fifth and eighth” for last year, he said.

Impressive numbers. Russo's tried a number of methods to improve education, including a few that would give secular left-wingers hissy fits:

“One of the things that have helped our children most is faith-based partnerships, which provide in-class help, tutoring and mentoring to our students. Right now, we have 27 churches involved. But we have more than 200 churches in Moore County, so we’re right at 12 or 13 percent. We need more involvement.”

Children, he said, “need mentors, role models and faith-based core values.”

It's not surprising that this would be an acceptable method of educational reform in the Deep South, and it's also not surprising that it appears to be working. Moore County has fewer than 75,000 inhabitants spread out over less than 700 square miles - but I still bet that the "200 churches" number is an underestimate. Assuming everyone in the county attends church, which wouldn't be unreasonable, that works out to one church per 375 inhabitants.

Another "discriminatory" exam in Delaware

Did you know that the state of Delaware is being accused of racial bias in the hiring of police officers during the 1990's? That's right, they're being sued for this, because in 1981 they had the nerve to implement a reading exam on which black and white applicant groups scored differently. Delaware discontinued the exam in 1998, presumably under pressure from activist groups who see any indication of group differences as proof of "bias":

The federal government contends the discrimination came through a written literacy test given to trooper applicants and the pass-fail cutoff levels Delaware used to measure results. White applicants regularly outscored black applicants on the test, given to screen admittance into the state police program. The state no longer uses it.

U.S. District Judge Kent A. Jordan ruled in May that the test adversely affected black candidates. That shifted the burden of proof to the state, which now is attempting to show the test was used lawfully with pass-fail cutoff rates that were job-related and measured minimum qualifications in reading and writing.

So, if you use a test on which every applicant group doesn't pass with the exact same rate, the pressure is then on you to prove that the test does indeed measure useful skills and is a minimum-competency exam. I believe that test developers should indeed be prepared to defend their exams in this way, but I believe this is a rotten precedent to set for challenging an exam. The test scores in the public school system show that minority students don't score as highly as white students, presumably because of their rotten schools. Why expect the test scores to be different at this level?

[Lt. Ralph H.] Davis testified that at the academy, where trainees are sent after a screening, future officers learn skills such as how to write paperwork requesting and justifying searches and arrests. The documents can be crucial to furthering an investigation and helping jog an officer's memory years later when it is time to testify.

Under cross-examination, Davis said officers recruited by municipal police organizations also attend the academy and sit side-by-side with state police recruits. They live in the same academy dormitory, do the same course work for the same instructors and take the same academy tests, he said.

Lt. Davis is attempting to justify the exams, presumably by showing that (1) reading skills are necessary in police work and (2) all recruits do the same course work and the same test preparation.

So let's find out more about this test. According to this website, written exams are still required for Delaware State Police, although it doesn't say what kind. Here's another article from earlier this year in which the allegations of bias are explained a little more fully:

Federal attorneys submitted test results from 10 recruit classes in court papers to demonstrate their argument. Percentages of passing black applicants dipped as low as 33 percent in one set of test results while white applicants passed at rates in the 80 percent and 90 percent ranges.

"The difference in passing rates is so extreme that it cannot be attributed to chance," federal attorneys wrote in court papers.

What has that got to do with anything? Why would black and white applicants be expected to differ only by chance? Look at the K-12 test scores, people. If black students don't get the reading skills in high school, why expect them to have them by the time they apply for the police force?

Yet another case of shooting the messenger. These results show that black students who earn diplomas or GEDs - the requirement for the trooper position - aren't being taught to read. So we're going to remove the test just so we can put black cops on the job who may not be able to decipher arrest warrants?

But attorneys for Delaware have attacked the federal argument, calling it "folly" bolstered by "invented facts," according to court papers. The federal statistics, they argue, were based on results for everyone who took the test, whether or not the applicants met minimum qualifications to be troopers.

That could certainly skew the results the way the federal government wants them to be. The use of reading scores of applicants with poor education or prior criminal history (who would have been rejected regardless) to claim test bias is despicable. There's an agenda being pushed here, and it's not one that will lead to a better police force.

The defense cited a different study of test results, which measured the performance of applicants who met job eligibility requirements. That study's results "recognized that blacks were fairly represented, or even slightly overrepresented, in the test-passer pool," according to court documents.

Emphasis mine. So, to recap, black applicants who met all eligibility requirements are fairly represented in those who passed the exams - but because blacks are over-represented in the failing group, and an unknown number of them would not have qualified to be a police officer on other grounds, that's evidence that the test is biased.

Give me a break.

(Thanks to Daryl for the tip.)

The MCAS lawsuit

The Boston Globe has the latest update on the lawsuit against the MCAS for alleged racial bias. The article begins with a sympathetic portrait of a black special education student who was denied a diploma because he did not pass the exit exam:

...sitting at his dining-room table one morning last week, William E. Lowe Jr. had much more immediate concerns than his ongoing lawsuit challenging the use of MCAS as a graduation requirement. The 18-year-old former high school football player was explaining why he has applied for a job at a local Blockbuster Video store instead of getting ready to study digital design.

Both federal and state judges have rejected their requests for preliminary injunctions to block the state from using MCAS to deny diplomas to the class of 2003, the first required to pass the test to graduate...In April, Suffolk Superior Court Judge Margot Botsford denied the state injunction request, commenting in her written opinion: ``[T]he plaintiffs have not shown a likelihood of succeeding in their challenge.''

One attorney believes the plaintiffs have little chance of winning:

"The chances of any court overturning a state test like MCAS are very low,'' said Boston attorney Miriam Kurtzig Freedman, who represents school districts and is author of "Testing Students ... And the Law." "The best they can do is tell the state to tinker with parts of it. But the bottom line is that states are entitled and in fact have a responsibility to establish standards. And the things that go against the students are the high number of passes and the fact that we have remediation'' through tutoring programs."

What's all the fuss about? Well, over 4000 students - 7 percent of the class of 2003 - have not yet passed the required MCAS, which measure English and math skills at the 10th-grade level. While that 7 percent number doesn't seem high to me, the plaintiffs are charging that the exam discriminates "against minorities, special-education students, and limited-English students, whom it says have not been adequately prepared for the exam."

To begin with, a test of English does not discriminate against those with limited English ability. It just identifies those who don't know the English language that well, which is what this test is supposed to do. The school is supposed to teach English, to everyone; those who don't learn it don't get a diploma. That's tough, but if a diploma is supposed to guarantee fluency in English, that's the way it has to be.

The other claims are based on the fact that minority and special education students pass at different rates than other students. This doesn't mean the test discriminates against them unfairly; if they don't know the material, then they don't pass the test. The real question to be asked is: Why haven't they learned the material? It may be that special education students may never learn it, so there should be (and are) options for those students. It's not that I don't understand the plaintiff's frustration if, in fact, students were not taught the material on the exam. But the protestors are wrong to focus their guns on the exam.

What's more, they shouldn't be fighting over high school students at all. Look at these 2003 third-grade MCAS results. Go to page 4. Only 63% of Massachusett's third-graders rate as Proficient based on the overall exam. The rate for "limited English proficiency" youth is, of course, much lower.

Then go to page 5. Seventy-one percent of White students are Proficient, but only 40% of the African-Americans, and 32% of the Hispanics, meet this same standard. If these kids are doing this poorly in the third grade, how can we expect them to learn what they need to learn in high school, exit exam or no exit exam?

Removing the exam doesn't fix the problem. The signs are there from the beginning. It's time to start fighting about, and finding solutions to, what will actually help minority kids learn English and math skills, starting at the beginning, and continuing on through high school. It's time for Massachusetts to start looking at elementary schools that do well, such as those that use Open Court phonics-based instruction or a Core Knowledge curriculum, and start imitating those schools.

Setting standards in Virginia

Devoted Reader Richard sent along a link to a Washington Post letter by one Susan O'Brien Saccomando, the chair of the science department at Mountain View Alternative High School. It concerns the Virginia Standards of Learning, or SOLs:

In the past few years I have heard complaints from just about everyone about the Virginia Standards of Learning (SOLs) and the accompanying assessment tests...But the complainers are wrong. The (SOLs) are working.

Before the SOLs, to pass a course, a student came to class regularly, participated, turned in class work and homework and took some tests...Effort played a big role in a student's grade.

The benefits of the SOLs are immense. Educated people make better citizens and more productive workers. Businesses can become partners with local schools and have a direct influence upon the workforce of the future.

In other words, states, schools within states, and even classes within schools all differ so much in terms of what is taught, how it's taught, and how it's graded. An A in Algebra II is going to mean something very different from school to school (as can be seen in the recent Fortier High fiasco), and standardized tests are one of the few ways - perhaps the only way - to assess whether all teachers are giving their students the basic preparation needed in courses. The tests are also a reaction to the student-centered, anti-fact, "naturalistic" teaching methods that have infested public schools since the 1960's, and these tests should be considered evidence that such a method, while it might be fun for a student, doesn't do much to prepare them to think critically using solid knowledge in the real world.

Parents demand FCAT Exam release

A parent-teacher coalition is agitating in Tallahassee for the release of FCAT exam booklets and answer sheets. They feel more scrutiny is warranted due to the high-stakes nature of the exam:

"Any test used to make life-altering decisions about children should be open to scrutiny," coalition president Gloria Pipkin said during a news media conference.

It's not likely he will do so.

And why not? Well, while groups such as "Free the FCAT" in Florida and "WASL Revolt: Rescuing Kids from Corporate Pedophiles" in Washington might have the right intentions (despite their vulgar and exaggerated names), they aren't cognizant of the financial impact of their requests. Private testing companies such as ETS can create new test forms for every administration because they've got the money and the staff to do so. States often don't have these resources, so they're forced to re-use test forms. Releasing those forms essentially makes them invalid for re-use, so the state either has to find the money and time to create and field-test new forms, or give old forms that will most likely produce a gain in test scores.

FCAT representatives make the claim that parents already received detailed enough information about how their children perform on the important constructs. If so, then it's hard to see this information-gathering movement as an attempt to benefit kids. More likely, it's an attempt to discredit the test, or demonize the test developers, in any way possible.

The parents are probably aware that NCS Pearson, who mistakenly denied diplomas to 7,000 Minnesota students, is the same firm responsible for grading the FCAT. I'm not saying that parents shouldn't be aware of these things, but it would be nice to see a journalist spend a bit more time pointing out just how impractical releasing every FCAT test booklet would be, or to see one challenge the assumption that refusal to release the test forms means the state has something to hide.

Of course, the usual blather about how it's impossible to create a standardized test that's not culturally biased is repeated here:

Marion Brady, vice president of FCAR and a former Brevard County school administrator who lives in Cocoa, opposes the FCAT's use as a high-stakes measure of student progress.

"I don't like the use of the test for such high stakes," Brady said. "It's impossible to write a standardized test that's not culturally biased."

What Ms. Brady really means is that it's not acceptable to write a test that requires all students to understand a core set of basic skills, and understand items about them that are written in Standard English. She's right in line with Senator Wilson, who previously said that the FCAT flunks students who can read perfectly well because it isn't "culturally sensitive". The racism inherent in these types of statements is clear if one examines sample third-grade reading items.

Just once I'd like to see a journalist question one of these crusaders as to where specific cultural insensitivity exists in the FCAT. Take for example Item 3, on page 27 of this document. Who does this discriminate against? What kid in Florida is unfamiliar with the idea that some scary creatures live in the ocean? If anything, you'd think a kid from Idaho would have trouble with this one.

The FCAT's final flunkers

School bells will soon be ringing in Florida, but for more than 1,300 students in Palm Beach, those bells will sound especially familiar. Those students flunked the FCAT in the spring - and flunked exams again after their stint at the summer reading academy - so they'll be repeating the third grade this year:

About 2,800 third-graders failed the reading test last spring. But more than 1,000 were able to proceed to fourth grade, thanks to exceptions to the rule that include passing alternative exams, being a native speaker of a foreign language or submitting a portfolio of work that proved they were working on grade level.

So, over a third passed through exceptions alone - something few reporters,and no testing critics, who cover FCAT controversies tend to mention.

Although some schools will have few retained students this year, others face more substantial numbers that require the schools to reorganize. Forest Park Elementary School in Boynton Beach, which will have 42 repeat third-graders, will add a third-grade class and distribute the some students among its other four third-grade classes.

"We have to rethink how we're approaching how they can be successful," Principal Bill Thompson said. "The old model of everyone getting the same amount of attention from the teacher and everyone having the same size class has not worked. Some students will have to get more attention than others."

Um, yes. Students learn at different rates and some will need extra help. If anything, I hope this moves schools away from the idea that students of differing abilities can easily be combined into one grade-level classroom. Tracking helps all students, while combining heterogenous students leaves the smart ones bored and the not-so-smart ones behind.

The Broadening of the American Test

Sunday's Boston Globe has an article on one crusader's attempts to change the SAT into a "broader, more varied" exam. How broad, you ask? Well, the College Board is apparently still stinging over recent criticism that the SAT measures too narrow a range of abilities, not to mention that inconvenient racial score gap, so they've commissioned various test critics to develop new additions to the SAT to measure qualities ''other than cognitive ability,'' according to VP for research Wayne Camara.

Qualities "other than cognitive ability." That phrase sets off warning bells in my head, and after reading the article, I realize the warnings are there for a reason:

Someday, if [APA President] Robert Sternberg has his way, college applicants across America will be judged according to their answers to a set of questions quite different from the ones they are used to.

What would you do if you had already eaten lunch when you realized you didn't have the cash to pay for it? Or: What would you do if you walked into a party where you didn't know anyone? Or: How would you ask a professor you didn't know well to write you a recommendation? Or: Write a story entitled ''The Octopus's Shoes.''

My, these read very much like...IQ test questions. In fact, they're very similar to WAIS-R (Wechsler Adult Intelligence Scale-Revised) Comprehension items. The items aren't widely available online, but cover topics such as 16 questions covering a wide range of problem situations involving such issues as health, social mores, social judgement, common sense, grasp of social conventions, interpersonal relations, and laws (e.g. “Why do people who want a divorce have to go to court?”). It should be noted that this component is but a small part of what makes up the overall IQ score.

While life might be easier for someone who knows what to do when confronted with a party room full of strangers, what in the Sam Hill does that have to do with collegiate academic abilities? Why should someone who can handle that situation (or can describe how they would handle it) be judged competent to take advanced English and calculus courses? Why is "common sense" being added to a test of cognitive abilities? What about when common sense and creativity are contradictory? (There are several things you could do to get a free lunch, but very few things you should do). And why should free-form creative writing be added when a test of writing skills is already scheduled for inclusion on the SAT in 2005?

Why add these squishy, controversial, subjective concepts to college admissions tests at all?

Why, to close the racial score gap, of course:

As for Sternberg, he says it's his job to invent a test that reflects a more nuanced view of intelligence - not, at this point, to sell that view to the American people. ''If we used the Rainbow Project, you'll find the kids at community colleges who do as well or better than the kids at Yale,'' he said...

''There are people who are really good at traditional tests, who may get 800s, and then when they get out of school, that's the end of the story for them,'' Sternberg said. ''They don't get along with people. They don't persuade people to listen to them.''

Funny, I haven't read about many people who make perfect scores on the SAT but lead terrible lives. I haven't read about many brilliant people who consider their post-collegiate lives to be "the end of the story." The urban legends of the brilliant but antisocial geeks tend to be just that - legends.

Does Sternberg have much data to support his theories? There's his work with the Rainbow Project, in which he attempted to measure creative ability by giving a test to 1,000 subjects. The examinees were asked to dictate stories, write stories, and write captions for cartoons. Points were given for "cleverness, humor, and originality." He allegedly found smaller score gaps on this test than on the SAT, and better prediction of college performances, but I'd like more specifics than this article provides.

How were the students recruited? What was their baseline performance on academic measures? Could they make huge spelling and grammatical errors and still get high scores based on creativity? Which colleges did they ultimately attend? How much smaller were the score gaps on this test than on the SAT? How long did it take to train the raters, and how reliable were their ratings (reliability being a measure of generalizability)? Training a few raters to judge "cleverness, humor, and originality" for 1000 examinees is one thing; training raters who can do this consistently for the 1.5 million examinees who take the SAT every year is another.

What's more, if questions like "If Y = 5 and X = 6, what is Y+X?" are accused of being racially biased, how are you going to design a key for the item, "What would you do if you walked into a party where you didn't know anyone?" that is acceptable to everyone? I would argue that, for this item, there are indeed multiple correct answers based on culture - so many, in fact, that I'd be hard put to say what "correct" means in this case. Could that squishiness be the reason for the reduced score gap; i.e., a wide range of responses were considered to be "correct"?

Notice, too, that the BG reporter skips over asking the obvious question, which is that if an SAT score gap exists, why is the focus on creating new tests that avoid that gap, rather than modifying the K-12 education so that the score gap closes naturally? Why does it make more sense to start measuring "creativity" than to make sure that minority kids learn to read earlier? Why is the focus on the ends, and not the means?

Not everyone is buying Sternberg's theories:

...Sternberg's endeavor is fraught with uncertaintly, not least because his questions challenge traditional concepts of intelligence.

Camara said Sternberg's questions read ''almost like a personality test'' and ''will have to pass the smell test'' with parents and educators. College administrators balk at the idea of requiring another test, which could put them at a competitive disadvantage in the race for applicants. And parents might wonder why their children's future was riding on unorthodox measurements such as skill in writing captions for cartoons.

And parents would be right to do so. If their kids attend public schools, they've been exhaustively tested on reading and mathematics basics, not creativity. If they've been attending private schools, they've most likely been enrolled in demanding academic classes and haven't had much time to waste on writing cartoon captions.

...Despite the calls for a more comprehensive test, said the College Board's Camara, it has been difficult to persuade universities to contribute to the massive endeavor of researching a new test. Then, when it's ready, will admissions offices, parents, and high school principals put credence in a test that asks students how they would deal with an unwelcome guest?

''The biggest issue about viability is not whether we think it's viable, it's whether colleges do it,'' he said. ''It honestly takes courage, it takes good science and good results, and even after that it takes one or two institutions which have a lot of courage to say `We're going to use it.'''

Better get to work on that "good science" part of this, Mr. Camera.

Joanne Jacobs sums it up:

The purpose of the SATs is to evaluate students' likelihood of success in college, not whether they'll do well in later life or be fun at parties. If judging the humor of their cartoon captions is a better predictor of college success than asking them to solve math problems, so be it. I can just see the SAT prep courses where students will drill in "tacit knowledge" and creativity.

Her points are good, especially the one about the creativity drills. If a test is high-stakes, then, regardless of its content, the creation of prep courses for it will be a lucrative business. The claim that a creativity- or common-sense-based test reduces the gap between rich and not-so-rich students will then become moot.

LEAPing into school grades

Standardized testing opponents who get upset when tests are used to make high-stakes decisions are going to be mighty unhappy with the Jefferson (LA) school parish, now that Jefferson is tying grades to test performance:

One parish school system has added more incentive for students to perform well on the standardized tests used by the state to judge public schools. Now the students who take such tests will see their scores factored into their own grades in subjects including English, mathematics, science and social studies.

"It has grown in majority opinion that this would bring students to really read the test, take the test seriously," Jefferson Parish superintendent Diane Roussel said. "And this would give us a more accurate reflection of what they need to know."

Whoo. Hear that kids? If you didn't take the tests seriously before, you had better do so now. But why do I get the hunch that this is meant to prevent grade inflation, rather than meant to motivate kids to do better?

And the critics are already complaining:

The plan has some critics, however, including C.C. Campbell-Rock, co-founder of the New Orleans-based group Parents for Educational Justice, which opposes the state's use of the LEAP to hold back some students.

"That's just adding more test to the testocracy we already have," Campbell-Rock said. "We now have a system where tests are driving the entire system."

As the president of the East bank Parents Advisory Council points out, the tests in fact will account for only 25 percent of the last-quarter grades, which translates to one-sixteenth of the grades for the entire year. Hardly a situation in which tests are "driving the entire system." And can we call a moratorium on the nonsense term "testocracy"? Testing critics rarely offer support for the claim that tests measure nothing except the ability to take tests, so ranking students by test scores is the same as ranking them by academic achievement or ability, which has always been a large part of our educational system.

On the other hand, though, it remains to be seen if LEAP scores are in fact useful or valid when used in this manner. If the LEAP content is in sync with the classroom content, then this might be useful - but it will be interesting to see any opposition that arises from the teachers who have to use it in this manner.

Bugging Governor Bush

Theyyy're back! Your favorite protestors! Those rootin', tootin', loony FCAT opponents! They've got the kids rounded up and protesting at the Florida Capitol Building, all because they believe the FCAT is unfair and should not be used for grade promotion.

Where do I start? First off, there's that photo. The third-grader in question looks pretty mopey already, although whether that's due to FCAT blues, or to the fact that activists roped him into spending a nice summer day inside a government building, it's hard to tell. And then there's that sign - "I am just nine years old. Please don't break my spirit." Gee, who knew that implementing testing to ensure that Danny gets taught to read in the third grade would be breaking his spirit?

And the inane quotes from FCAT opponents just keep coming:

The noon rally was organized by Sen. Frederica Wilson, D-Miami, a former educator who said she has been visiting summer reading programs to document what she considers the harmful and unnecessarily rigid enforcement of FCAT reading requirements..."They can read. They just can't pass a high-stakes test that is not culturally sensitive to them," Wilson said.

Hear that, Danny? Sen, Wilson believes that you do not have the proper culture to learn how to take a standardized test. She does not believe you are capable of understanding this exam, with its questions about a boy named Peter writing letters from a cat named Peaches, unless it can be modified to fit your "culture." If you recognize this for the racism that it is, then I say you're bright enough to be promoted into fourth grade.

Wilson's other comments are more meaningless than stupid. For example, we "must keep in mind that not all children are good test-takers," and that "these children are becoming collateral damage." Not once does she offer a reason for why some kids might be not be good test-takers (other than the wholly-unsupported "cultural" reason), nor does she mention the long list of exemptions that allow students to be promoted to fourth grade despite a failing FCAT score. She'd rather parents and readers assume that the FCAT is an absolute barrier, which it is not.

Oh, and there's her comment that, "You hold students back on the basis of professional-judgment reasons...You don't retain them on the basis of a test that comes out of a computer." What the computer part of it has to do with anything, I don't know, but I assume she means to contrast the cold, unfeeling world of testing and computers with the warm, fuzzy world of teaching. A dangerous comparison, considering that teaching third-graders to mistrust computers is almost as destructive as failing to teach them to read.

Walking the walk, talking the talk

By now, I'm sure you've all heard about the hapless school superintendent in Lawrence, MA, who failed to pass a literacy test that is required of all teachers. Still, I can't help but take a shot at this story myself...

Superintendent of Schools Wilfredo T. Laboy, who recently put two dozen teachers on unpaid leave for failing a basic English proficiency test, has himself flunked a required literacy test three times, The Eagle-Tribune reported Sunday. Laboy, who called his failing scores ''frustrating'' and ''emotional,'' blamed a lack of preparation and concentration, and his lack of English skills. Spanish is his first language.

Interesting, isn't it, how his "lack of English skills" is the last thing listed here as his reasons for flunking? And why are we supposed to be sympathetic to his "frustration" when he's placed teachers on unpaid leave for flunking the thing? He should have been the first to take this test, if only to judge for himself whether or not the test was valid.

State Education Commissioner David P. Driscoll said he is aware of Laboy's troubles with the test, but would not say how many chances Laboy would be given to pass or what the consequences of another failure could be. He commended Laboy on an ''excellent job'' leading the district, but said ''he's going to have to pass.''

Sounds like those teachers who claim that kids are doing "excellent classwork," while also flunking the basic skills exams. However, perhaps I'm being unfair. Perhaps leadership for this district doesn't require much in the way of literacy skills, which is a scary thought indeed.

Since 1998, all educators from teachers to superintendents have had to pass the Communications and Literacy Skills Test, which measures basic reading and writing skills, including vocabulary, punctuation, grammar, spelling and capitalization. Laboy barely passed the reading section on his second attempt, scoring the minimum required grade, he said this week. He also failed the writing portion three times, and a section requiring test-takers to transcribe a passage read over an audiotape, using proper punctuation and spelling.

Ouch! Is this the test from hell, or does Laboy really need to brush up on his English skills? The test objectives are here, so let's see for ourselves.

The subtests are Reading and Writing - that's pretty straightforward. "Determine the meanings of words and phrases." "Understand the main idea and supporting details in written material." "Recognize effective sentences." We're not talking rocket science, or tricky testing mumbo-jumbo here. We're talking figuring out what the main point of a passage is, or recognizing a run-on sentence.

And here's the test information booklet. Scroll down to the bottom for the sample multiple-choice and free-response items. The items aren't all easy - certainly, defining "democracy" is harder than defining "abolish" - but the items still should be within reach of anyone with a college degree, especially someone who is working in a school superintendent position.

''What brought me down was the rules of grammar and punctuation,'' Laboy said. ''English being a second language for me, I didn't do well in writing. If you're not an English teacher, you don't look at the rules on a regular basis.''

However, if you use English on a regular basis, you should be familiar with the rules. This "But-I'm-not-an-English-teacher" excuse sounds familiar. It's what many adults use when they're caught making flagrant errors in English usage. It's obvious that they believe only English teachers should be required to write a coherent sentence and make themselves comprehensible. The excuse doesn't carry much weight with me, as you might have anticipated.

Laboy is in a directorial position in our public school system, and receives a salary of $156,560 a year for doing so. Is it too much to ask that such a person be able to pass a test of English literacy, and also not be allowed to hide behind the twin excuses of not being able to concentrate and not being an English teacher?

Update: Joanne Jacobs has not one but two postings that are peripherally related to this topic. The first is a link to an article by National Review author Rich Lowry, who wonders why these bilingual teachers can't pass English literacy tests. My guess is that superintendent Laboy considered himself "bilingual" as well, but he flunked an English exam. So did 22 out of 25 "bilingual" teachers in his town.

In the second posting, Joanne relates the story of Kiet Tran, a hapless teen-age Vietnamese immigrant who was enrolled in a Madison, WI high school. He was promptly placed in "bilingual" class taught entirely in Spanish - and his English-speaking father was not able to get Kiet transferred into English-speaking classes. The family finally had to move out of the district in order to get English classes for the boy.

Rich Lowry's comment that "...bilingual education is a misnomer. It is really monolingual education, in any language but English." is beginning not to seem like an exaggeration. When immigrants have to fight to be taught English, and teachers who are incompetent in English sue to keep their "bilingual" teaching positions, something is very, very wrong.

Bashing the SAT - update

The New Orleans Times-Picayune has quite a different viewpoint about those infamous SAT analogy items than does the LA Times:

...the College Board's decision to drop the analogy section from the verbal portion of the test is to explicable as Adam Sandler is to tolerable.

Test makers should of course take pains to eliminate regional and cultural biases from the SAT and other standardized tests. But it's not at all clear that those differences won't show up in the new sections of the test -- or that it's the College Board's fault if they do...

With this last statement, the article sneaks up on the important acknowledgement that group score differences are not necessarily indicators of test bias, althought it would have been nicer if the writer had made that explicit. The LA Times writer, on the other hand, seemed pretty sure that it is the College Board's fault if there are group differences, as was evident by the writer's willingness to give plenty of ink to test critics who use the word "bias" loosely and incorrectly.

Bashing the SAT

You know, I'm still feeling under the weather, with low energy and even lower mood. But Devoted Reader Bas Braams knows what will get me up and shaking my fist at the world - SAT criticism that includes obsolete items. This LA Times article drags up, yet again, the infamous but long-gone "regatta" item as part of current criticism against the test:

...the National Center for Fair & Open Testing...contends that the SAT is biased against lower-income students and those for whom English is a second language. As evidence, center public education director Robert Schaeffer cited several analogy questions from over the years, including this one, since deleted:

"That's incredibly culturally centered," Schaeffer said. "You don't see a regatta in center-city L.A., you don't see it in Appalachia, you don't see it in New Mexico."

I've emphasized the phrase "over the years" above, and for good reason. It's an attempt to drag up old news and disguise it as current criticism. This item hasn't been on the SAT for at least 13 years, perhaps longer. For at least that same amount of time, SAT items have undergone review for differential item functioning in order to root out these types of items - something the writer doesn't mention here.

According to Schaeffer's reasoning, because a few SAT items were once perhaps biased, we should assume that most of them still are. What 's more, no evidence is given here to show that at the time the item was written and used, it was in fact biased. If the word "regatta" was once on vocabulary lists, then it could have been unbiased, and there's no shame in the College Board deleting items once they're obsolete.

What Schaeffer is really saying is that now it isn't politically correct to insist that disadvantaged kids learn the word "regatta," and it's odd to see that taken seriously as test criticism.

Schaeffer also doesn't seem to understand the very basic purpose of the SAT, which is that it measures English comprehension. Therefore, kids who don't know English well should do worse on the SAT than kids who do know English. Yet, in his mind, this is "bias" against non-native English speakers, rather than evidence that the SAT is a valid measure of how well a kid knows English. Funny how test critics use the word "bias" so loosely, isn't it? ETS and the College Board should put the word "bias" in an analogy item so that the situation will come full circle.

Sadly, this article is somewhat commendable because writer Paul Pringle at least hints that the offending item is no longer in use. Some writers don't even go that far, as can be seen in this article; an unsuspecting reader might believe that this item was still in use.

(By the way, there is a regatta in New Mexico. Just so you know.)

I'm also not impressed by this tired argument:

...the SAT is often the target of complaints that material like the analogies can be professionally coached. Many say that gives an edge to students whose parents can afford tutors and prep courses.

The analogies, he said, are spiked with "distracters" — words designed to trip up students. An example: A question begins with the prompt "COVEN: WITCHES." The possible answers include "amulet: vampires," but the correct choice is "choir: singers." "The student is distracted by the superficial relationship between witches and vampires," Berger said.

What? You mean the SAT is a test for which one can actually prepare? Horrors! I eagerly await an alternative test from these critics - one on which preparation affords no advantage, so that a kid who practices under time limits and familiarizes himself with the tasks will do the same as a kid of similar aptitude who is completely unprepared for the occasion. After all, we wouldn't want to reward extra effort, would we?

Oh, and that "superficial distractor"? That's, um, the point. An amulet is not composed of vampires. The occult reference in this response makes it an attractive distractor, but a kid who understands the relationship won't pick the wrong option. I suppose Berger's ideal test item would be one in which all but the key response are clearly wrong. So - the test should not reward practice, and all correct responses should be very obvious. Have I got all this straight?