Check out the new guy
Michael Lopez has a blog - Highered Intelligence. He caught an article about the SAT that I missed, and blogs it well, with the proper mixture of humor and frustration. Give him a try.
Speaking of that SAT article, I think the CSM is on-target with the comment, "The ultimate problem lies in the poor quality of many public schools. If colleges can help uplift those schools, then just changing the SAT may become less important."
The Princeton Review gives its (unsolicited) opinion
As an employee of a non-profit testing company, I have my reservations about the big for-profit test preparation companies such as Kaplan and the Princeton Review. Yes, they have apparently helped many, but at a steep cost, and the modest score gains that testing companies report are at odds with the large gains trumpeted by the test-prep companies' press releases. Plus, while some test-prep companies pride themselves on educating the test-takers, other companies claim all standardized tests are meaningless and attempt to teach their clients how to cheat or "game" the tests.
So, it was a surprise to see that the Princeton Review has apparently decided to evaluate testing programs across the country. Programs that were given a low mark, such as Oregon, are understandably ticked off, with a member of the Oregon Department of Education claiming that their poor marks are because the Princeton Review "are a publicly traded for-profit company that we don't do business with."
The report is 46 pages long, so here are some highlights.
Good stuff:
*The general concept. Yeah, yeah, I know, psychometricians and test prepsters are normally at cross-purposes and don't usually have much good to say about each other, but I'm glad to see a project like this come up, and PR is honest in saying that this is their first attempt, and that the results should be interpreted cautiously.
* PR did the web research and provides a list of websites for each state in Appendix III - worth downloading for that source alone.
* Some of their indicators were solid - there should be substantial item overlap between what's in the standards and what's on the test, for example. And all of the indicators for Criterion #3, save one (see below) are good.
* They provide the raw data, which is good, because they didn't explain their scoring/scaling/number-assignment/ranking procedures in a clear or straightforward way, so I'd like to take a different pass at the raw numbers.
Bad stuff:
* I read, tucked away in a footnote on page 2, that PR is now providing tutorial services for Massachusetts (test improvement), Hawaii (applicant outreach) and Washington, DC (SAT prep). This suggests a lack of journalistic or scientific distance during the preparation of this report.
* The indicators are not well-explained in the appendix. There's 1a, "Are standards granular enough so that a small number of test items can reasonably meaure a student's mastery of skills?" I assume this means the standards should be more fine-tuned than, "Student should be able to read", which is true. However, they give 2 points - the maximum - if specific statements can be measured with "single items", but then the example is a specific skill ("Add two-digit numbers") So I don't know what they mean. Surely schools don't get higher ratings for setting standards so fine that one test item accompanies each standard.
* Also, what does PR mean by saying an item is "well-written"? And who is the someone other than the "test developer" who should be reviewing the item - do they mean other test developers, or QC staff, or outside reviewers, or what?
* Now that I think about it, the 0-2 scoring method is not the best scale, I think, for what they are trying to measure, because 0 can mean the school doesn't satisfy the indicator or doesn't make the information public, which really leaves only a rating of 1 or 2 to distinguish among schools who are making some effort. You need more "granularity", guys.
Iffy stuff
* The report is honest about the fact that states received scores of "0" for information that PR couldn't find, but that still seems dishonest.
* PR should proofread more carefully. Twenty-five indicators scored 0-2 means a max raw score of 50 points per school, but on page 2 we read "The mid-ranked state, Utah, receive a raw score of 130 out of a possible 200, while the ten bottom-ranked programs all had scores of 97 or lower." Just one typo, I know, it should be "scaled score", but you can't take programs to task for poor quality and implementation and then make a mistake on the second page that will leave the average reader wondering if they've already gotten lost.
* The Criterion #3 indicator that I disagree with - PR thinks contract terms with testing companies should be public knowledge, but I was not aware that any such contract is public knowledge, or that it should be.
* Ranks, and grades, and raw scores, and scaled scores, all on the same data, are too much, but that's just my opinion.
"Way-off" scores
CNN has the gruesome news: The standardized testing company Harcourt Inc. confesses to scoring errors in the Georgia Stanford Achievement Tests. CNN doesn't provide a lot of information on exactly how many test scores are in error, or what the company plans to do about it. There are no big decisions, such as graduation or admission, related to these scores, so it doesn't appear that test-takers need to be compensated in any way. Errors like this make the news for the same reason that plane crashes draw big headlines - so many tests are given each year without errors that it is legitimate news when the quality control processes fail at a testing company. However, it's more ammunition for the anti-testing crowd, who expect the testing industry to be error-free.
Eating my words...
You know, I often find myself defending my field in social situations. I go to a party, other partygoers ask what I do, I say, "I'm a psychometrician", then I get blank looks, then I explain what that is....and every so often I get someone who's still pissed off about a bad testing experience. So they proceed to use me as a soundboard to rant and rave - by God, here is a real live person to blame for their testing ordeal. Most people are humorous about it and have some sense of perspective, although I understand how they can still be needled by a monitor who cheated them out of five minutes of testing time or a computerized test that shut down in the middle of their time slot. It's frustrating, and the test-taker doesn't have much control over any of the process. However, sometimes someone will get so out of hand with their complaints that I find myself saying, "Dude, calm down. It's not a matter of life and death."
I never thought I'd be eating those words.
Now, however, it looks like a one-point difference on an IQ test is going to be a matter of life and death. The recent Supreme Court decision that capital punishment of the mentally retarded is "cruel and unusual punishment" is all over the news. There are reams (or whatever the internet equivalent is) of commentary on it - in the Washington Post, by Jonah Goldberg in the Jewish World Review, Lowell Ponte in FrontPage magazine, Don Feder in the Boston Herald, in Newsweek, by Brian Doherty in Reason Online...and so forth. So much has been said about it that at first, I wasn't sure what I could add.
Even if I'm just repeating here what others have said, though, I do have to get my two psychometric cents in. My first thought was, oh, so NOW you think an IQ test is a great test, eh? None of this "emotional intelligence" or "multiple intelligences" or "IQ isn't everything" or "IQ scores are culturally biased" nonsense anymore, now that, as several commentators put it, a low IQ score is a Get Out of the Chair Free card. Yes, mental retardation is generally defined as an IQ of less than 70, but I hope that the definition of mental retardation that could keep someone out of the chair would be based on more than a test score. After all, researchers who use IQ scores to predict certain variables or who suggest that IQ scores should be used to make policy decisions (especially if their viewpoints are not particularly politically correct) often hear cries of outrage that such tests do not predict the whole person, that such tests don't tell you anything about motivation, or desire, or ambition, or potential, or "true" intelligence. So, why can't a prosecutor claim that when someone with an IQ of 65 murders another human, their actions speak louder than their test scores, and they should be judged on those actions?
It's not that I don't understand the desire to show compassion for those who, due to lack of intelligence, have been coerced into confessing or who were not able to adequately defend themselves against some serious charges. I think that death penalty cases should never hinge on just a confession, and that these cases should face greater or more careful scrutiny than they currently do. But setting a cutpoint for IQ scores and excusing everyone below it does not fall in the category of more careful scrutiny. It's extremely careless, in fact, to the point of being meaningless. IQ tests were not designed to keep people from being put to death, and if they are heavily relied on for selecting that outcome, we'll gradually move towards the time when only someone with a college degree, or some other obvious past testament to intelligence, will face the death penalty.
I also find it interesting that I didn't see mentioned anywhere just how many mentally retarded inmates get put to death each year - is it 10%? Less than 1%? Are the inmates in the 20 states that currently do not have laws against executing the mentally retarded going to clamor for retests, or will their current IQ scores stand? The potential for abuse is astounding here, and I wouldn't want to be the clinician in charge of testing individuals, knowing that a difference of a few points is indeed a matter of life and death. What's more, what is there now to stop the argument that incarcerating mentally retarded criminals for life is also cruel and unusual? Are we showing compassion here, or are we on a slippery slope to deciding that no punishment is acceptable once a certain IQ threshold is crossed? What about those with the "low-normal" IQ criminals - those in-between 70 and 90? Certainly, they're viewed as capable of functioning, albeit at a lower level, but should they receive less punishment, or less severe punishment, than the brilliant criminals? Is there now going to be an IQ-punishment continuum? Do only the Ted Bundys of the criminal world deserve to sit on the hot seat?
Hmmm, I don't seem to have any answers here, only questions.
Whew!
I'm finally back, having traveled halfway 'round the globe to attend one testing conference, then came back and visited my alma mater for yet another conference. In case you're interested, the two conferences were the International Conference on Testing and Psychometric Society. I've been given the honor of being the editor for the PsycSoc website, so if you have any commentary, please send it along.
Man, it's good to be back. Thanks to those of you who were patient enough to write in even though the site had not been updated for a while.....
Disappearing again...
I'm off again for another business trip. I'll try to answer all the emails before I leave but if I don't, rest assured that (a) I've read all the email and (b) I'll get to it all when I get back, because I got a lot of thoughtful (and argumentative) emails that deserve replies. My apologies.
In defense of memorization
Claudia Winkler of the Weekly Standard restates a common-sense argument about the importance of memorization, which is out of vogue in today's educational climate. After all, how can we expect kids to ever learn those higher-order, critical thinking skills without having a base of the "rote memorization" of facts and specifics to build on?
Hot Rod magazines can raise boys' reading scores
Here's a delightful article about one way to raise the reading scores of boys, who consistently lag behind girls. John Scieskza set out to rectify the "Reading is for girls!" mindset by starting an initiative called Guys Read (guysread.com). He gleefully reports that the voters on the site "have recommended everything from Hot Rod magazines to Phillip Pullman's award-winning His Dark Materials fantasy to Louis L'Amour Westerns to Katharine Paterson's Bridge to Terabithia to some of my favorite Terry Pratchett science fiction novels and beyond." If you're the parent of a boy who doesn't like reading or isn't interested in the selections offered by schools, go check it out.
Is this "censorship"?
The Nando Times has an indignant article about the "censorship" of literary selections used as New York Regency Exam passages. It seems Nando found the exam's sensitivity review process to be heavy-handed - literary passages that were selected for use on the exam were altered to remove references to "race, religion, ethnicity, sex, nudity and profanity", often without the consent of the original author. Reuters has a similar story here.
Now, I'm a free-speech advocate, but I'm not siding with Nando on this topic. The main error the test developers have made here is that they didn't inform the authors of the chosen passages that changes might be made, and I find it surprising that these authors didn't sign a contract agreeing to the use of their works in the original or in an altered form. The New York State alumni assistant commissioner cited in both articles emphasizes that "the 'fair use' provision of copyright law allowed the excising of passages for testing purposes."
Regardless, the test developers are right to remove as much potentially controversial material from the reading passages as possible. Yes, it can alter the meaning of the text; yes, it can make the reading experience less enjoyable. That's preferable to the alternative, which is putting kids in a high-stress situation and testing them on material that may be distracting or emotionally disturbing enough to interfere with their performance. This is not the same situation as teaching emotionally intense material in class (where I feel that altering or ignoring certain texts would indeed be censorship). Controversial reading material should be taught slowly and carefully in a classroom, so I don't see the benefit of presenting it suddenly and out of context in a high-stakes testing environment. A few indignant authors are nothing compared to the potential lawsuits from test takers.
What tests would the school psychologist use?
In the process of doing some online research for a concerned parent today, I found this site - What tests would the school psychologist use? I think it's pretty useful as a starting point, if you know your child is going to be tested but you have yet to get any details.
Parents might also find it useful to obtain a copy of the Standards for Educational and Psychological Testing. This reference book is meant for psychometricians and test administrators, but parents may be interested to have this as backup information. The chapters on general principles of test use and clinical assessments may be of particular interest.
Test results for bilingual education and English immersion
The National Center for Policy Analysis reports that language immersion is three times more effective in producing English fluency than bilingual education. California and Arizona parents have already rejected bilingual education, which seems more likely to strand kids in a linguistic ghetto, and Massachusetts parents will soon have their say.
Home again
Finally back from Business Trip #1 (of 3 this month). I suppose I shouldn't complain that my company put me up in a luxurious resort in the Caribbean, but it was a bit of a pain that I had no reliable internet access this week (you sympathize, I'm sure). Anyhow, I'm just now getting to the many emails that I got from those of you who found my site via Fox News. Thanks for writing (and thank God that not all of the 400 or so of you who found me this week decided to write). I'll get to you as quickly as I can.