Number 2 Pencil

Inputs vs. outputs

Reg Weaver, the head of the nation's largest teachers' union, rails against the current focus on "output":

On Monday, Weaver balked at focusing on students’ standardized test results. “In many instances they want us all to focus on outputs,” he said. “An output is nothing more than a test score, and as long as they get us focused on a test score, then they cause the public and many legislators not to deal with the inputs.”

“Inputs,” he said include not only class size and adequate funding but qualified and certified teachers, safe and orderly schools and state-of-the-art technology.

Is the public really not interested in inputs? Or have they had it with inputs, like state-of-the-art technology and fancy teaching programs, that might be pricey but haven't been shown to be related to those important outputs? I don't think any parent would not want a school to be more safe and orderly, but they might balk at laptops for all if the school still doesn't teach kids how to read.

Weaver [also] offered general advice. “We can begin to introduce legislation that talks about closing the loopholes,” he said and added that corporate tax relief totals billions of dollars in lost income in some states. “We can begin to make sure that Arkansas school districts have what they need... if these children are going to be able to be successful. And let me tell you that if in fact the achievement gaps that exist here are going to be closed, it is not going to be done with charter schools, and it is not going to be done with pay for performance.”

Why not? Certainly, merit pay for teachers who do well could certainly be considered an important "input." And if a parent decides to pull their kids out of the public system and enroll them in a charter school, my guess is they're thinking about a lot of inputs, like discipline and safety and class size and textbooks.

A loophole in NC

North Carolina gives new teachers an out:

Legislators and North Carolina state education leaders reached an agreement Monday over how to ease teacher licensing rules and allow local schools to hire more out-of-state candidates. A 26-member panel of lawmakers, teachers and administrators agreed to recommend making it easier for teachers with less than three years of experience to receive permanent licenses as long as they earn positive evaluations from supervisors.

How did they make it easier? By changing one thing -

The final step for a state license has been requiring a teacher to pass a standardized test or complete a standardized evaluation program. Monday's recommendation would create a third option: a positive evaluation from a supervisor that reflects the teacher's ability to "impact student learning" and that the school district has offered to continue employing the teacher.

Despite all the hooha about how this "expands the options" of school districts and "emphasizes a standard" for new teachers, all they've done here is make it possible for a teacher to be offered a permanent position even if they can't pass the Praxis II and specialty exams, and satisfy all the administrators who need to fill teaching slots and aren't worried that these new folks can't pass these tests.

The Kayak Kid takes on the Ohio exam

Hoo-boy. A smart Ohio student, John Wood, takes a very public stance against an exit exam:

My high school graduation took place during the Memorial Day weekend. However, despite being ranked sixth in my class, I did not cross the stage that day, and my dad, our high school principal, did not give me a diploma. I did not drop out at the last minute, and I was not expelled. I didn’t graduate because I refused to take the Ohio Proficiency Tests.

I did this because I believe these high-stakes tests (which are required for graduation) are biased, irrelevant, and completely unnecessary.

For whom? Irrelevant for those who are ranked sixth in their class and have a principal for a father to boot? Unnecessary for those who won't have much trouble grabbing a GED and some online time to air their anti-testing views?

The bias of the tests is demonstrated by Ohio’s own statistics. They show consistently that schools with high numbers of low-income and/or minority students score lower on state tests. It is argued (in defense of testing) that this is not the tests’ fault, that the scores are only a reflection of the deeper socioeconomic injustices. This is very likely true. What makes the tests biased is the fact that the state does little or nothing to compensate for the differences that the students experience outside the classroom. In fact, the state only worsens the situation with its funding system. Ohio’s archaic school funding system underfunds schools in poorer areas because it is based on property taxes. The way we fund our schools has been declared unconstitutional four times, and yet the state legislature refuses to fix the problem.

Tests that measure what we intend them to measure - in this case, academic performance - are not biased. Critics have quite the tendency to stretch the definition of the word "bias" into anything they see as "not quite fair," and that's what John is doing here. What is happening here is "differential impact," which is something else altogether (see here for a previous discussion). What John is fighting against here is the right of the state to use a test that has differential impact - a test showing that students whose parents don't contribute much financially towards education tend not to do so well. He obviously doesn't believe that parents should have a choice in deciding where their property tax money goes. He should label his fight as such, rather than erroneously tossing around words like "bias."

The irrelevance of these tests is also demonstrated by state statistics—in this case, the lack of them. In 13 years of testing, Ohio has failed to conduct any studies linking scores on the proficiency tests to college-acceptance rates, college grades, income levels, incarceration rates, dropout rates, scores on military-recruiting tests, or any other similar statistic.

If true, this proves only that Ohio has failed to produce data showing the relevance of the exams. It does not prove that the tests are irrelevant. What's more, if the purpose of the exam is to show that a student has learned what they need to know in high school, then we would not expect the test to be any more predictive of variables such as income, incarceration, or college grades than we would expect high school performance to be. If achieving a high-school diploma is not highly correlated to college grades, there's no reason to expect the test scores to correlate with those grades - but that's not proof the test is irrelevant.

More important, a system already exists for determining when students are ready to graduate. The ongoing assessment by teachers who spend hours with the students is more than sufficient for determining when they are ready to graduate.

If it weren't for grade inflation, social promotion, widely-varying standards among teachers, and the fact that teachers nowadays are unlikely to flunk the vast numbers of students who deserve it, he might have a point here.

in southeastern Ohio, alternative assessments are alive and kicking. At my school, Federal Hocking High School, in Stewart, Ohio, every senior has to complete a senior project (I built a kayak), compile a graduation portfolio, and defend his or her work in front of a panel of teachers in order to graduate. These types of performance assessments are much more individualized and authentic, and are certainly difficult, something I can attest to, having completed them myself.

(1) The key word here is is "individualized," which can translate to, "Almost impossible to use to compare students over time, students within schools, or schools to one another." Good for some students, bad for seeing whether large groups of students are improving.
(2) Does John have data showing that kayak-building skills are highly positively related to college grades or income? No? Then why does he claim the Ohio tests are irrelevant based on the lack of that same information?
(3) A kayak? In Ohio? The point was?

Note that John is already signed up to attend college this fall, and his voice - in the form of a well-written article - has now been heard publicly. He obviously has not been disadvantaged by this situation, either from the exam itself or his failure to take it. Does he really believe, I wonder, that kids who enter high school still struggling with reading and basic math would benefit as much as he did from a kayak-building experience in place of a basic skills exam?

Update: I forgot to mention that this week's EdWeek has a forum set up to discuss this article. Be sure to leave a comment.

Update #2: Excellent discussion by Quincy, including a link to Joanne's comments about the heartbreak of discovering in college that you aren't quite as great as you thought you were in high school.

Be sure to blame the tests

I'm confused. When droves of students fail a particular test, why is the default headline one that criticizes the test, rather than the students' knowledge?

If this year's results are any indication, some struggling students may want to dust off their textbooks this summer.

Almost one-third of [Georgia's] eighth-graders failed the math portion of the [Criterion Referenced Competency Test] and another 17 percent flunked the reading portion. About a quarter of the students failed the science portion and another 20 percent failed the Language Arts section of the exam.

"Nobody is kidding themselves," said Dana Tofig, spokesman for the state Department of Education. "The eighth grade scores were not where anyone wanted them to be."

What's the next quote the reporter dug up? Three guesses, and the first two don't count.

A similar poor showing by eighth-graders next year could put thousands of students at risk of being held back from high school. And lackluster performance in the test also could drive needy pupils away from the school system, said Merchuria Chase Williams, president of the Georgia Association of Educators.

"It's a high-stakes test. And to us, it's a high-stakes test at its worst," Williams said. "One single test is used to determine whether or not a child has learned when, in fact, a teacher's evaluation is not even taken into consideration."

Quite frankly, when a teacher fails to teach students well, why should we care about their evaluation? Let's look at sample items from the math portion, failed by approximately 33% of Georgia's eighth-graders, shall we?

See anything there that seems horribly unfair or ambiguous? See any reason to suggest that students who fail this portion are somehow ready for high school math? No? Let examine the sample items for language arts. See anything too bizarre or tricky for 13-year-olds?

It is odd that there seems to be a dip in results here; last year's results were more promising for eighth-graders, while this year's results are better for the lower grades. The Atlanta Journal-Constitution suggest the middle school curriculum is squarely to blame.

Regardless, is it too much to ask for articles about standardized tests to include substantive criticisms of those tests, given that at least some criticism seems to be required? Something other than, "This is high stakes and we don't like that"?

Having an impact

Betsy's Page catches a telling comment in Thomas Friedman's NYTimes article about Williams College, which asks graduating seniors to name the high school teachers that had the greatest impact on their lives. Betsy noted this quote:

When she got the call from Williams saying she had won, [nominated Highland Park High School teacher] Ms. Loris recalled, "I just kept saying, 'Wow.' " A teacher for 23 years, now nearing retirement, she added, "I just found it very affirming in a Zenlike way," an acknowledgement "that my days have value, my life has had some worth. Public school teachers don't get that very often," especially with No Child Left Behind restrictions, which now require teachers to teach to the tests, and push out the window "all those things that really spark kids imaginations" - like art and music.

Betsy is outraged:

What a gratuitous slap for Ms. Loris and then for Thomas Friedman to include. Come on, teachers at Highland Park High School aren't losing their classes in art and music for NCLB preparation classes. If you check out HPHS's test scores, they're doing just fine for the great majority of their students. However, the groups that aren't passing Illinois's required 11th grade standardized test are the economically disadvantaged, disabled, and Hispanic students. Would Ms. Loris object to money spent to help those students achieve more in reading, writing, and math? The fact that this school has a teacher in international relations is some indication that NCLB has not forced this school to make draconian cutbacks. Plus, I object to the idea that art and music are the only things that "really spark kids imaginiations." I think that reading books can also spark those imaginations and, if kids have low reading skills, not only will they miss out on that spark to their imaginations, but they'll be held back in school and in work for the rest of their lives.

I can't improve on that, except to say than anyone laboring under the twin delusions that (a) all that matters about education is that it fires one's "imagination", and (b) subjects like reading and math can't possibly excite that imagination, shouldn't be in the field of education.

Ignore my test scores, look at my plant photos

Another day, another demand that we stop measuring academic abilities in school and start assessing the "whole" child:

Every year, millions of children in California are required to take a standardized test known as the STAR test. I am dreading the thought of taking this test in the spring and I know I’m not the only student who feels this way...

The results of a STAR test do not measure the most important traits of a student. For instance, they don’t reveal anything about the child’s moral development. Is the student happy, compassionate or kind? Does the student enjoy learning and school?

Note: The author here claims that the most important traits of a student are not academic achievement. Instead, that is outranked by enjoyment of learning, and happiness. Pop quiz: Who do you think will be happier as an adult, in the real word - the student who learned to be "happy" in school, or the student who developed a solid foundation in reading, math, science, etc?

For that matter, why is the author assuming these are mutually exclusive? Wouldn't it be fun to produce research showing that the students who learn the most in school and do the best on standardized tests are also the ones who are happiest and have the most love of learning? I'm not saying I know that's so; I'm saying it would be fun to poke at the anti-testing folks with those kinds of correlational results.

One alternative to the STAR test is called portfolio assessment. This is where the teacher and student work together to collect samples of work throughout the school year. These samples are kept in a "portfolio" or large folder. The teacher uses the portfolio during conferences with parents and the student, and the teacher reviews the work in the portfolio to check student progress.

Portfolios are time-consuming, and I thought teachers were always harried and pressed for time. Bias is an issue - what if your teacher doesn't like you, or like kids like you? How can this be compared across different classrooms, never mind different schools? What if the teacher's feedback is subpar, so that the assessments don't improve over time? What if this year's teacher is an easy grader, and next year's teacher is not?

Last year, my teacher did a project with the class called Reflections/Connections. This project was a good way of showing individual student progress. My teacher asked each child to complete a different assignment for each subject. We could choose to create an art project, a piece of writing, make a video or invent some other way to fulfill the requirement. For instance, I planted a garden, photographed the growth of the plants and created a scrapbook of all my photos along with notes about each plant for science. I thought it was interesting to see the different ways each kid expressed him/herself. This project revealed much more about our individuality.

Isn't that precious. As we all know, individuality is the most important thing. Did this author learn the scientific names for plants? Did she do a valid research study to see which plants grew the tallest, and why? Did she write a summary that demonstrated her understanding of the science involved? If she did, did every other kid in his class do as well? Or was individuality and effort the only thing graded here?

Please note: I know the author is a sixth-grader. For a kid that age, she writes well. And I know that class projects like this can be informative as well as fun. I am not so much ragging on the budding botanist as I am on the Santa Cruz Sentinel staff, who apparently believe it is newsworthy that a student enjoys planting a garden more than taking a standardized test.

What minority parents want

Some radical educators in NYC might be surprised to learn that the groups they supposedly speak for don't agree with them:

Last week, a group called the New York Collective of Radical Educators staged a protest against standardized testing. Responding to recent reports about substantial gains for fourth-graders on citywide reading and writing examinations, the group argued that the improved scores reflect "drill-and-kill" test-preparation activities rather than real learning. Worst of all, protesters maintained, the entire testing enterprise discriminates against racial minorities. For blacks and Hispanics especially, they said, standardized tests inhibit academic achievement and increase the dropout rate.

The only problem is, blacks and Hispanics don't see it that way.

Over the past decade, public opinion surveys have demonstrated overwhelming support among racial minorities for high-stakes testing. In a 2003 study by the Pew Hispanic Center, for example, three-quarters of Latinos said that standardized tests "should be used to determine whether students are promoted or can graduate"...Likewise, African-Americans favor high-stakes tests by large margins. To be sure, activist groups like the National Association for the Advancement of Colored People have criticized NCLB and state graduation exams. But the black rank and file tell another story.

Read it all. While it is true that both sides could be correct on some points, it is rare to see the minority parent pro-testing attitudes mentioned in the debate on race and testing. Unfortunately, I could see an anti-testing activist reading this article and taking away the message that they should redouble their efforts to convince parents that all tests are somehow racist.

The author of the article above, Jonathan Zimmerman, also wrote Whose America? Culture Wars in the Public Schools. NYCore, the group he mentions, lists their talking points here; you won't be surprised to learn that they don't believe in letting parents support local schools with their property tax money, white men aren't exactly welcome in their organization, and military recruiters are the evil embodiment of US "imperialism."

CNN vs. the tests

Recently, CNN aired a special on NCLB and the current status of testing. I didn't get the chance to see it, but Education Gadfly, who did, wasn't impressed:

While brief moments are devoted to explaining how standards and testing can turn around schools, the many teacher diatribes against NCLB and student woes from standardized testing make it pretty clear where America's most trusted name in news stands. The program insinuates that the problems with the Houston school district miracle/myth will be replicated around the country as a result of NCLB. In describing the cheating that occurred, the blame falls not on cheating teachers but on former superintendent Rod Paige and his "reign of terror." In fact, in true [Michael] Moore fashion, the program seems to suggest a Paige/Bush cabal to fake achievement, win the presidential election, and force testing on the unsuspecting nation.

While there are plenty of sad student tales, missing are the stories of those hurt by the old system or helped by the new one. As Manhattan Institute's Jay Greene notes (the program was basically Greene vs. everyone else), "Any system [will] create some sad outcome for somebody," and while just giving everyone diplomas "might help some students, you would hurt many more. And that kind of system is rotten, and it's produced the stagnation that we've had for the last three decades." More disturbing is that this slanted special is a "Classroom Edition" intended to be shown to students. The internet workbook for teachers (see here) asks such questions as, "What do you think are some possible 'unanticipated' social, political, economic or psychological consequences that could occur as a result of high-stakes testing and mandatory retention?" Not that we're telegraphing our punches, mind you.

I clicked on the internet workbook, and let me assure you, the Gadfly isn't exaggerating. There are 10 "Discussion Questions" listed, and three are about the negative impact of testing and mandatory retentions. One is about alternatives to testing, one is about cheating, one is about why this is called a "battle," and one wants the students' "reactions" to claims about dropout rates. There are three questions (# 2, 9, and 10) that I consider reasonably balanced and not provocative, but as I said, I didn't see the program. Given the Gadfly's review, it sounds like CNN was hoping for negative responses to questions such as, "After watching this program, what questions do you have about NCLB and high stakes testing?"

Testing with no consequences

An op-ed in the Amarillo Globe News misses the point (annoying free reg required):

Tests don't test how much you really know but rather how much you know about what's on that particular test on the day you take it.

How this could be construed as a critical comment is a mystery to me. Does the author have some other option in mind that, perhaps, measures how much students will know in the future? Is there some magical alternative to tests that shows how much students will know next week? And is there any kind of assessment in the world that measures something other than what's actually on the assessment? Given that the asssessment can be made as broad or narrow as desirable, it seems odd to use this as a criticism.

Teachers must teach to tests or risk lower scores, which will ultimately affect the school's standing and the teacher's job.

Let me get this straight. Teachers must teach their students the material that their employers, the school/district/state, has deemed important, and students must demonstrate that teachers have done their jobs by performing well on exams. Otherwise, the state can conclude that the teachers aren't doing their jobs. Does anyone see a problem with this? Does anyone see why the role of "teacher" should be defined in such a way as to be mutually exclusive of the tasks above?

Some percentage of the student population can express what they know more fully using media other than standard tests; e.g., written essay, oral defense or demonstration.

Of course, some of those methods are so unreliable and difficult to grade properly that one could argue that, unlike with most tests composed of multiple-choice items, some students gain higher scores through these methods purely due to measurement error.

Oddly, the article then moves from the "criticisms" above to the perfectly reasonable argument that testing followed by no corrections, or by doubling inefficient efforts, are not useful. I just find it difficult to understand why the author spends several sentences bashing tests, and then moves to:

...what companies don't talk about, and probably couldn't even if someone asked, is how they use the results of tests to improve their operations. This qualitative distinction separates the successful from the unsuccessful. Those who use test results to highlight the need for action tend to do better than those who use them to punish.

But I thought tests don't actually measure how much you know. If the tests are that flawed, why do we expect that schools who use them to "highlight the need for action" will do better than schools who don't? Was the test-bashing in the first part of the article obligatory for someone seeking his anti-NCLB credentials? Or, like many test-bashers, is this critic merely trying to redefine tests (a la his call to abolish standardized tests in his conclusion) so that schools can use them only for feedback on individual students, and not to do those unhappy comparisons from school to school, or state to state?

An argument for testing

Op-ed writer Brent Staples says in the NYTimes what I have been saying on this blog for quite some time:

The same civil rights groups that sing hosannas to Brown have been curiously muted - and occasionally even hostile - to No Child Left Behind. But the groups have mainly been missing from the debate...Why are civil rights groups standing on the sidelines instead of fighting to ensure that this law succeeds? The reasons are numerous and complex...

[one of the reasons is the] antitesting argument. Civil rights activists commonly embrace the popular but erroneous view that the reading and math tests associated with No Child Left Behind are culturally biased or unfair to minority children. Paradoxically, those who hold this view are often middle- and upper-class African-Americans who have law degrees and Ph.D.'s, which require rigorous tests and high achievement.

The simple achievement tests required under the law are essential to the objective of closing the education gap. By arguing that these tests are inappropriate and culturally biased, these members of the liberal black elite have unwittingly embraced the worst stereotypes about the poor. They have also given cover to politicians who believe that the achievement gap can never be closed and that minority children can never reach the levels attained by their white, affluent counterparts.

In other words, the claim that tests of basic reading and math skills are biased against all minorities is itself a bigoted and racist argument. For some reason I've never understood, many of those who oppose testing unthinkingly stand behind a notion that the KKK would be proud to claim as their own: that multiple-choice questions are somehow innately impossible for non-white students to solve.

Smarter testing opponents will claim that the test is fine but the use of it is not, or that the differential impact of the test scores is the problem. Sometimes, this argument is correct, but even these critics stop too soon in the logical process. If a graduation exam does in fact have a differential impact for minority students (fewer of them pass, so fewer graduate), one must go beyond the scores and ask why. Too many critics merely suggest removing the tests, as opposing to eradicating the problems that cause minority students to perform poorly. Too many critics assume that everything must be wrong with the tests, and nothing wrong with the teaching.

Getting it right or wrong

Parents march against the WASL (Washington Assessment of Student Learning):

Shelley Anderson keeps photocopies of her oldest son's test scores from the past six years in a black folder crammed with statistics about the Washington Assessment of Student Learning. "I dislike what this is doing for our children. I think it's setting them up for failure," she said Monday. "It's teaching them to think inside the box. I have three children, and they're distinct individuals."

Anderson brought the black folder with her from Spokane to Olympia, where she took part in a protest against the WASL, which is being given to students across the state this week.

I don't doubt her children are individuals. However, it's hard to understand why requiring fourth-graders to read on this level, or requiring seventh-graders to comprehend math on this level, is "setting them up for failure."

As expected, many of the complaints are that students who do well in classes do poorly on the exam, which leads one to wonder why parents aren't protesting ineffective classroom scholarship or inflated grading schemes. But at least one parent compares the WASL scores to other test scores:

Anderson pointed to the scores of her oldest son, Bill. He failed the math sections of the WASL in fourth and seventh grade but passed the Iowa Test of Basic Skills in sixth and ninth grades with above-average marks.

Bill Anderson, 15, noted that students receive partial credit for following the right procedure in the math section, even if they answer with the wrong number. "It's not set up to test you on what you know; it tests you on how you think," he said.

According to this website, partial credit scoring is indeed used on the ITBS. While that's all well and good, it's wrong to say that partial credit scoring is testing "how you think," while all-or-none scoring is only testing "what you know." If you don't know how to think your way through the problem, you won't know the answer. And if you don't get the right final answer, there's obviously a step missing in your thought processes.

I don't think partial-credit scoring is a bad thing, but it's just plain to silly to say that it's qualitatively different, and better, than right-or-wrong scoring. Partial-credit just makes it easier for the judges to see where someone went wrong (which is why partial-credit scoring is often used in the classroom environment), and it makes it easier for students who don't know how to arrive at the final answer to gain at least some credit for a not-totally-wrong answer, which is why it would be more popular with people who oppose standardized testing.

Protesting the TAKS

Here are all your littlest testing opponents in one article:

So far, Macario is the only student in Texas whose public boycott against the TAKS tests puts him in immediate danger of not being promoted to the next grade. But he's not alone in his complaints. A handful of bright, outspoken Texas kids, including some from Haltom High School, are protesting what they say is the decline of education since TAKS' arrival in 2002. Grass-roots student protests similar to those in Texas have taken place around the nation.

"I hope they make people listen. I know I am," says Rep. Dora Olivo, D-Rosenberg, a Texas legislator who says learning centers are turning into testing grounds.

I've discussed Macario's situation before, as well as Kimberly Marciniak's opposition. Unlike most school administrators, I don't find the anti-testing t-shirts to be disruptive. But I do feel that students who feel this strongly about the exams should be prepared to suffer all the negative consequences that come from skipping out on them.

I also have to note that all the students who are protesting the exams seem to be good students who are most likely bored by these tests. I wonder if they're really thinking about the positive consequences the exams can have for students who are struggling and stuck in poor schools, who would benefit from a renewed focus on basic skills.

But you can judge people by their arguments

An op-ed about the SAT, "You can't judge people by their scores", that misses the boat entirely:

An interesting study recently emerged from Bates College, in Lewiston, Maine. Bates is one of nearly 400 colleges and universities (according to FairTest) that no longer require standardized tests for admission. At these schools, applicants have the option of submitting their SATs. Admission officers have long assumed that applicants who choose not to submit their scores have lower scores than those who do. This proved true. In fact, there was a considerable difference -- 160 points -- between the two groups. So far, no surprises, right?

So, now for the interesting part. The study showed no difference in academic performance or graduation rates between students who submitted their scores and those who did not. It also found that there was little difference among SAT submitters and non-submitters and their chosen career paths, with the exception of fields that require additional standardized testing, such as graduate programs in medicine, law and business.

So, what does this all mean? It clearly strengthens the argument that the SAT and ACT are not accurate predictors of either intelligence or of potential for success in college and afterwards. High-school grades, course choices and class rank are far more precise indicators.

No, it clearly does not.

It supports the argument that the SAT may not be highly related to college GPA or graduation rates or career paths for Bates College. If there's one thing I'd like opinionists to get straight before I die, it is the concept that the SAT's predictive validity can vary from college to college. It might not be useful for Bates, but one cannot generalize from that scenario to say that this is proof it's not useful elsewhere. Universities differ, standards differ, and students differ. The validity of an exam does not exist in a vacuum, and "the predictive validity of the SAT," as a concept or value, can be interpreted only in how it is used in a particular situation.

What's more, there are plenty of reasons that the SAT might not be correlated with GPA at Bates, some less flattering to Bates than others. Grade inflation may be rampant, and a reduced variance in GPA would necessarily result in a reduced correlation with the SAT. A 160-point mean difference is large, but not huge, and this certainly doesn't mean that someone with rock-bottom SATs will do just fine at Bates. What's more, this little caveat is particularly weaselly:

...It also found that there was little difference among SAT submitters and non-submitters and their chosen career paths, with the exception of fields that require additional standardized testing, such as graduate programs in medicine, law and business.

I find it fascinating that, of all the ways in which these fields could differ from the other programs at Bates, the one this writer mentions is that these fields require further testing. True - and they also require solid scientific knowledge, good math skills, top-notch reading ability, good memorizational and organizational skills, etc. Gee - I wonder if maybe the students with the higher SATs are more likely to end up in these fields because they're better qualified?

This writer wants schools to re-evaluate the SAT's usefulness, but perhaps, "even better," says schools should drop the SAT altogether. Why would it be better to drop an admissions step without doing the research, than to do the actual research? Is that perhaps because this writer really doesn't want anyone to know how useful the SAT is at some universities?

Update: Illuminaria crunches the numbers. Read it all.

Protesting the lack of fun

Note: I've moved this post to the top after adding an update. Newer posts are below.

We've got another little testing opponent on our hands:

Saying too much emphasis is placed on the Texas Assessment for Knowledge and Skills exams, fifth-grader Macario Guajardo is refusing to take this year’s TAKS reading, math and science tests.

"In fourth grade, I was under a lot of pressure for the TAKS, and I decided I wanted to something about it," he said. "Teachers focused on TAKS, and it wasn’t fun for us anymore. Sometimes we had to stay in from recess to prepare for it. It was a lot of worksheets."

Ah, yes. A product of the "schools must be FUN" generation.

With his parents’ permission, the student who makes As and Bs skipped school Wednesday to miss the reading test. He declined to take a makeup exam Thursday. State law says students must pass TAKS tests to pass each grade level. In cases of TAKS failures, a committee of parents, teachers and a principal must decide whether the student can advance.

Macario, 11, has already appeared in articles in The New York Times. Network TV news stations including NBC and Telemundo have also contacted his parents and school officials. Macario said he hasn’t received any bad vibes from teachers or students and that many seem to support him.

Wow. So he's definitely learning that there's pretty much no downside to opposing the tests, no matter how hysterical the arguments get about how tests are taking over our lives and being crammed down our throats.

Although he grew up hearing his father’s complaints about TAKS, the protest idea was entirely Macario’s, he said.

Uh-huh. Forgive me for being skeptical about this part, which seems to reflect only the trendy notion that children develop entirely apart from any parental influence whatsoever.

Macario’s parents are behind him 100 percent, said Francisco Guajardo, assistant professor in the University of Texas-Pan American’s educational leadership program. "He stopped having fun at school. For me, it just broke my heart to see my son frustrated, upset, even angry," he said.

Ah. His dad is a prof in an educational program - and regularly berates the TAKS. At least the dad is honest enough to admit that his views have affected his son:

Guajardo said he probably influenced Macario’s opinions. "We’ve been talking about this since Mac was a little baby. Mac grew up with this," he said. "I like the TAKS. … I’m not in support of the way it’s done. The whole school is organized around … the test. Teaching to the objective on the test — that’s pervasive (in the area"...

TAKS scores don’t improve graduation rates or SAT scores, Guajardo said...Teachers have little time to encourage students in imaginative, creative work. Macario hasn’t once had an art class in his elementary years, he said.

So who's really protesting here, I wonder? And does the lack of art really have anything to do with the TAKS focus? I'm thinking Guajardo knew a newsworthy story when he saw one.

Not that Guajardo Junior shouldn't be allowed to protest; heck, I give him credit for it. But I'd be more impressed if his argument was something other than school isn't fun enough for him anymore. The Bernard Chapin chapter I quoted earlier today addresses this issue as well:

There are a plethora of pseudo-scholars in the field of education that wholeheartedly approve of the videogameification of the modern classroom. Those “scholars” would have been disappointed to discover that our principal’s devotion to fun was more based on her own lack of seriousness than it was rooted in any educational methodology.

Update: Education Gadfly found another young testing opponent - interestingly, also in Texas...

Anti-testing types have taken up the cause of Mia Kang, a 14-year-old Texan who defied teachers and counselors and turned in a little essay announcing her opposition to standardized testing instead of completing a mandated practice TAKS test. She has vowed not to participate in the real thing this spring, even at the risk of not graduating from high school. Kang is one of a gaggle of Texas students who has refused to take state tests, and posters to the liberal blog Daily Kos hope to start a letter-writing campaign to ensure she will graduate despite opting out of the test.

Further, it's a strange form of civil disobedience that demands both notoriety for breaking the law and exemption from the consequences of law-breaking. If Mia Kang doesn't want to take the TAKS, fine. If someone's conscience dictates that they cannot participate in a mandated activity, they should refuse. But civil disobedience without consequences is merely showboating. Martin Luther King, Jr., wrote his magnificent "Letter From a Birmingham Jail" because he accepted the consequences of his refusal to accede to unjust laws. The nation was moved by his example to correct an injustice. "Letter from My Living Room" likely would not have had the same effect. And if Ms. Kang believes the TAKS to be unjust, we invite her—and would applaud her gumption in so doing, even if we disagree with her interpretation of the facts—to convince the Texas legislature of the rightness of her cause.

Good point.

Give us money, and don't ask what we do with it

Eduwonk notes that the national PTA viewpoint on testing doesn't exactly come down on the side of sharing educational information among parents:

The National PTA opposes:

PTA also has a new poll out about NCLB and what parents want. Except Eduwonk can't locate the poll, only the highlights in a press release. Even those are not a slam dunk for the anti-NCLB crowd...when the NEA buys a national interest group, don't they expect them to stay bought? Is there a warranty?

Note that the PTA opposes "federal mandates" for testing, or for comparison of schools by testing. Sounds like they want the federal government out of the schools, doesn't it?

Not really:

National PTA

* Supports annual passage of federal appropriations bills containing adequate levels of funding for education and child-related programs;
* Opposes funding proposals and budget process changes that cut or negatively impact the availability of funds for education and child-related programs;
* Opposes tax credits and deductions for elementary and secondary school tuition and other education-related expenses for public and nonpublic school students; and
* Supports public funds for public schools only and opposes using tax dollars to finance education vouchers for private and religious schools.

Got that? Everyone's money must go to public schools, but the government - and parents - can't demand accountability in the form of test scores in return.

A grand example of missing the point

A classic example of missing the point in a Daily Californian editorial:

Anyone who has had to endure standardized testing in high school knows how useless it is. It’s true there has to be some standard for education nationwide, and that California’s secondary educational system is in shambles when it comes to consistently turning out students with basic skills. But such standardized exams force teachers to “teach the test,” compromising the real learning that should be going on during class time. Those who refuse to bend to such tactics are forced to devote days, even weeks “teaching” students how to pass the test right before it’s thrown at them.

So the author admits both that there should be a nationwide standard, and that California high schoolers aren't learning basic skills. Yet the most efficient method of assessing an objective standard - standardized tests - are rejected, as are any attempts on the part of teachers to teach the basic skills measured by the tests. Somehow, "real learning" that won't be assessed must be imparted to these students who have missed on out basic education.

Of even more concern is student equity in taking these tests. Having standardized tests only favors the affluent—as soon as a new test is devised, test-taking programs will spring up, offering complete success on the exam for exorbitant prices.

Here we go again. The tests get blamed because test prep companies - who have never shown good evidence that they are worth the money - jack up their prices. The tests get blamed despite the fact that any student willing to stick to a schedule can prepare themselve using old tests for cheap (or free, if they go to any library). The tests get blamed despite the fact that students from affluent backgrounds do better in every sphere of educational development.

If the price of test prep companies are such a sore point, why don't we see editorials calling for outlawing the companies, rather than banning the tests?

When "an educational dream" doesn't have much to show for education

An op-ed in the Seattle Times says schools shouldn't be judged on test scores alone:

From the inside, Seattle's Orca elementary feels like an educational dream.
The school has energized parents, a passionate staff and a racially and economically mixed student body. It's popular, with waiting lists to get into kindergarten and first, second and third grades.

From the outside, though, something looks wrong at the South End alternative school. According to the state's bellwether standardized test, Orca is one of the worst schools in the Puget Sound region. Last year, only two of 37 Orca fourth-graders passed all three sections of the Washington Assessment of Student Learning (WASL). The school was last among Seattle's 70 elementary schools in writing and near the bottom in reading and math.

Predictably, the author then goes on to say it's "insane" to judge Orca by WASL scores. But is it really? The school may be popular, especially with parents who consider racial and economic diversity a selling point for a school. But if the teachers are doing such a great job, why are the kids not doing well on basic-skills tests?

It is troubling that Orca kids do so poorly on the tests. As Principal Ben Ostrom says, it raises the question of whether all students get enough "rigorous academic challenges."

But standardized tests don't work equally everywhere. One example: When Orca kids take in-class tests, sometimes they can choose to take them verbally instead of in writing. You can't do that with the WASL.

Well, no, you can't, because writing is one of the skills measured on the WASL. Why should we assume that a student who chooses to take tests orally (as one of my readers points out, the word "verbally" is incorrect here for making a distinction between written and spoken responses) is in fact learning what they need to know when it comes to writing? Why should we assume that it's perfectly okay for students to be tested in whatever manner they choose? If they learn better that way, fine, but they might be handicapped later on if they haven't learned to write well. And that's why Washington State tests that particular skills. And it explains why Orca is dead last in that area.

I'm sure the WASL isn't perfect. But to judge from the sample test items, it also isn't anything that should be throwing eager and educated young minds for a loop. If Orca's fourth-graders can't read an article about grey whales and write down two details from the article, there's a problem, and the blanket statement "standardized tests don't work equally everywhere" just doesn't cut it.

And neither does a bad attitude:

Last year, five fourth-graders boycotted [the WASL]. As 19-year Orca teacher Liz Neuman said:

"I could give a rip about the WASL. We can teach to this test, but then how much of what makes this school great will be left when we're done?"

Something tells me Ms. Neuman's insistence that learning to read stories, and write details about them, has nothing to do with a great education is part of the problem.

The logical fallacies of testing critics

Recently, Devoted Reader Adrian sent me a link to Nizkor's list of logical fallacies. If you're unfamiliar with Nizkor, it's a site that is hard to describe with mere adjectives, although I think a combination of "awe-inspiring", "courageous," "immensely admirable," "humbling,"emotional," "phenomenal," and "we need more people like this" would do for a start.

Essentially, Nizkor's mission is to combat Holocaust revisionists, which makes the work I do seem pretty insignificant. I first discoverd the site about eight years ago, and I had forgotten about their logical fallacies pages. Nizkor is pretty adept at parrying logical flubs, the kind often made by anti-Semitic "scholars" in their attempts to convince us that, gee, we didn't really lose all those millions of Jewish souls in the 1930's and '40s. Or if we did, it was an accident - but if it was deliberate, hey, maybe Hitler was on to something.

You know the type of slime I'm talking about. (The kind who got his ass handed to him by Deborah Lipstadt, that's who.)

Anyway, since Nizkor deals with this crap on a daily basis, they kindly present Dr. Michael C. Labossiere's Fallacy Tutorial, and to make a long story short, my Devoted Reader Adrian noticed that one variant of the "Appeal to Authority" fallacy seemed awfully close to the "fundamentalist reporting" I noted a while back. To put it shortly and sweetly:

Appeal to an Unnamed Authority. This fallacy is also known as an Appeal to an Unidentified Authority.

This fallacy is committed when a person asserts that a claim is true because an expert or authority makes the claim and the person does not actually identify the expert. Since the expert is not named or identified, there is no way to tell if the person is actually an expert. Unless the person is identified and has his expertise established, there is no reason to accept the claim.

As in, "Critics say tests are biased toward minorities." Simple, to the point - and wrong.

So this got me thinking. It's pretty easy to spot the other fallacies of testing critics that are mentioned on the Nizkor site:

* "Early psychometricians were white men, so they must have been racist." (Ad Hominem fallacy.)

* "Most teachers oppose standardized testing, so it must be wrong." (Appeal to Belief and Biased Sample fallacy.)

* "This standardized test upset an elementary school student, therefore it is wrong." (Appeal to Pity fallacy, at which Michael Winerip is an expert.)

* "I don't take tests well, so there's no way the SAT could predict my college grades." (Relativist Fallacy.)

* "It was in the news this week that there was a scoring error on the PRAXIS; ETS must make a lot of those errors." (Spotlight fallacy.)

* "You're a psychometrician, so of course any argument you make in support of testing must be taken with a grain of salt." (Circumstantial Ad Hominem, not to mention surreal.)

But I wonder - are there any other fallacies out there, not on Dr. Labossiere's list, that are more specific to testing critics? I think so, but I want some input from my readers as well.

Here are a few that I've thought of already:

* The "Live By the Statistics, Die By the Statistics" argument.

Evidence suggest X cannot be true, thus, Y must be true regardless of evidence.

This occurs when testing critics argue the inappropriateness of using a standardized test for predictive purposes, allegedly because the correlation of the test score with the dependent variable is "too low," but then suggest alternatives (such as interviews or essays) with no corresponding data to show that these alternatives are better predictors (as demonstrated here). This seems like a twisted alternative to the Burden of Proof fallacy; because testing critics have (they believe) provided proof that a test is not good enough, this relieves them of any obligation to provide proof that the alternatives they suggest are any good.

* The "Emotionally-charged Yet Undefined Word" fallacy.

X is true, even though no one knows what X is.

The obvious example here is bias, a word which is used in every article critical of standardized tests, yet is rarely properly defined. On the other hand, virtually every textbook on bias in test items presents, in the first chapter - nay, the preface - the definitions that psychometricians use when discussing bias. These terms aren't hard to find, and we don't hide what we think it means. But testing critics are very skilled at keeping exactly what they think it means a secret.

* The "800-Pound Gorilla In the Room" fallacy.

The cause of A must be anything other than what is most awkward to admit is the cause of A.

This is related to the Confusing Cause and Effect fallacy, in which one assumes that because A and B regularly occur together, A is the cause of B, and the Post Hoc fallacy, in which A occurs before B, therefore A must be the cause of B. But in the testing critic version, even when A and B always occur together and A always predates B, it must be true that A cannot be the cause of B. This happens when someone observes that, for example, poor teaching based on ill-defined concepts and "progressive" ideas often predate poor test scores, yet testing critics will claim that home life, discipline issues - indeed, anything except the curriculum - must be the cause of the low scores. It hardly needs to be said that this is also related to the Wishful Thinking fallacy.

* The "Omniscient Observer" fallacy.

Item X was created for Person A. Person B cannot solve Item X; therefore, Item X is not appropriate for Person A.

I'm thinking here of the logical fallacy that led reporters and observers to assume that because Governor Bush (who hasn't taken geometry in 30 years and doesn't use the stuff in daily life) couldn't answer an FCAT geometry item on the spot, he has no right to insist that Florida's high-schoolers take the test. In other words, testing supporters (or those who impose tests) must be content experts for any test they support - which would rule out a lot of parents.

Any others you can think of? Feel free to suggest new names for the ones I've already thought of, too; as you can probably tell by my post titles, I'm not so hot at coming up with pithy little blurbs.

Fundamentalist reporting

You know how I'm always carping about when lazy reporters start a sentence with "Critics say..." for the sole purpose of rehashing anti-testing myths? "Critics say the tests are biased." "Critics say the tests are not fair to minorities." And so on.

Well, Orson Scott Card (yep, the sci-fi author), writing for the Opinion Journal, is peeved about the very same phenomenon, which he catches in a credulous article about the eco-scare movie, "The Day After Tomorrow":

The whole point of this article is to make sure that the people who read it take "The Day After Tomorrow" far more seriously than the film deserves. Why? Because global warming has become one of the weapons used in the political war to bring down Western civilization, and without necessarily realizing it, the left-biased news media are completely buying into that political agenda...

This is the postreligious equivalent of a fundamentalist preacher starting a sentence with "The Bible says." It invokes authority without context, without understanding, and without admitting the possibility of error. (Most self-respecting fundamentalist preachers would at least tell you which book in the Bible they were quoting.)

Emphases mine. It's so nice to see that I'm not the only one driven mad by reporters who use intentionally undefined sources to make wild, sweeping statements about complicated issues. And I'm not surprised that a successful sci-fi author was able to describe the issue in much more complete, descriptive, and colorful a fashion than I ever could.