July 16, 2004

The logical fallacies of testing critics

Recently, Devoted Reader Adrian sent me a link to Nizkor's list of logical fallacies. If you're unfamiliar with Nizkor, it's a site that is hard to describe with mere adjectives, although I think a combination of "awe-inspiring", "courageous," "immensely admirable," "humbling,"emotional," "phenomenal," and "we need more people like this" would do for a start.

Essentially, Nizkor's mission is to combat Holocaust revisionists, which makes the work I do seem pretty insignificant. I first discoverd the site about eight years ago, and I had forgotten about their logical fallacies pages. Nizkor is pretty adept at parrying logical flubs, the kind often made by anti-Semitic "scholars" in their attempts to convince us that, gee, we didn't really lose all those millions of Jewish souls in the 1930's and '40s. Or if we did, it was an accident - but if it was deliberate, hey, maybe Hitler was on to something.

You know the type of slime I'm talking about. (The kind who got his ass handed to him by Deborah Lipstadt, that's who.)

Anyway, since Nizkor deals with this crap on a daily basis, they kindly present Dr. Michael C. Labossiere's Fallacy Tutorial, and to make a long story short, my Devoted Reader Adrian noticed that one variant of the "Appeal to Authority" fallacy seemed awfully close to the "fundamentalist reporting" I noted a while back. To put it shortly and sweetly:

Appeal to an Unnamed Authority. This fallacy is also known as an Appeal to an Unidentified Authority.

This fallacy is committed when a person asserts that a claim is true because an expert or authority makes the claim and the person does not actually identify the expert. Since the expert is not named or identified, there is no way to tell if the person is actually an expert. Unless the person is identified and has his expertise established, there is no reason to accept the claim.

As in, "Critics say tests are biased toward minorities." Simple, to the point - and wrong.

So this got me thinking. It's pretty easy to spot the other fallacies of testing critics that are mentioned on the Nizkor site:

* "Early psychometricians were white men, so they must have been racist." (Ad Hominem fallacy.)

* "Most teachers oppose standardized testing, so it must be wrong." (Appeal to Belief and Biased Sample fallacy.)

* "This standardized test upset an elementary school student, therefore it is wrong." (Appeal to Pity fallacy, at which Michael Winerip is an expert.)

* "I don't take tests well, so there's no way the SAT could predict my college grades." (Relativist Fallacy.)

* "It was in the news this week that there was a scoring error on the PRAXIS; ETS must make a lot of those errors." (Spotlight fallacy.)

* "You're a psychometrician, so of course any argument you make in support of testing must be taken with a grain of salt." (Circumstantial Ad Hominem, not to mention surreal.)

But I wonder - are there any other fallacies out there, not on Dr. Labossiere's list, that are more specific to testing critics? I think so, but I want some input from my readers as well.

Here are a few that I've thought of already:

* The "Live By the Statistics, Die By the Statistics" argument.

Evidence suggest X cannot be true, thus, Y must be true regardless of evidence.

This occurs when testing critics argue the inappropriateness of using a standardized test for predictive purposes, allegedly because the correlation of the test score with the dependent variable is "too low," but then suggest alternatives (such as interviews or essays) with no corresponding data to show that these alternatives are better predictors (as demonstrated here). This seems like a twisted alternative to the Burden of Proof fallacy; because testing critics have (they believe) provided proof that a test is not good enough, this relieves them of any obligation to provide proof that the alternatives they suggest are any good.

* The "Emotionally-charged Yet Undefined Word" fallacy.

X is true, even though no one knows what X is.

The obvious example here is bias, a word which is used in every article critical of standardized tests, yet is rarely properly defined. On the other hand, virtually every textbook on bias in test items presents, in the first chapter - nay, the preface - the definitions that psychometricians use when discussing bias. These terms aren't hard to find, and we don't hide what we think it means. But testing critics are very skilled at keeping exactly what they think it means a secret.

* The "800-Pound Gorilla In the Room" fallacy.

The cause of A must be anything other than what is most awkward to admit is the cause of A.

This is related to the Confusing Cause and Effect fallacy, in which one assumes that because A and B regularly occur together, A is the cause of B, and the Post Hoc fallacy, in which A occurs before B, therefore A must be the cause of B. But in the testing critic version, even when A and B always occur together and A always predates B, it must be true that A cannot be the cause of B. This happens when someone observes that, for example, poor teaching based on ill-defined concepts and "progressive" ideas often predate poor test scores, yet testing critics will claim that home life, discipline issues - indeed, anything except the curriculum - must be the cause of the low scores. It hardly needs to be said that this is also related to the Wishful Thinking fallacy.

* The "Omniscient Observer" fallacy.

Item X was created for Person A. Person B cannot solve Item X; therefore, Item X is not appropriate for Person A.

I'm thinking here of the logical fallacy that led reporters and observers to assume that because Governor Bush (who hasn't taken geometry in 30 years and doesn't use the stuff in daily life) couldn't answer an FCAT geometry item on the spot, he has no right to insist that Florida's high-schoolers take the test. In other words, testing supporters (or those who impose tests) must be content experts for any test they support - which would rule out a lot of parents.

Any others you can think of? Feel free to suggest new names for the ones I've already thought of, too; as you can probably tell by my post titles, I'm not so hot at coming up with pithy little blurbs.

Posted by kswygert at July 16, 2004 06:40 PM