Sunday's Boston Globe has an article on one crusader's attempts to change the SAT into a "broader, more varied" exam. How broad, you ask? Well, the College Board is apparently still stinging over recent criticism that the SAT measures too narrow a range of abilities, not to mention that inconvenient racial score gap, so they've commissioned various test critics to develop new additions to the SAT to measure qualities ''other than cognitive ability,'' according to VP for research Wayne Camara.
Qualities "other than cognitive ability." That phrase sets off warning bells in my head, and after reading the article, I realize the warnings are there for a reason:
Someday, if [APA President] Robert Sternberg has his way, college applicants across America will be judged according to their answers to a set of questions quite different from the ones they are used to.
What would you do if you had already eaten lunch when you realized you didn't have the cash to pay for it? Or: What would you do if you walked into a party where you didn't know anyone? Or: How would you ask a professor you didn't know well to write you a recommendation? Or: Write a story entitled ''The Octopus's Shoes.''
My, these read very much like...IQ test questions. In fact, they're very similar to WAIS-R (Wechsler Adult Intelligence Scale-Revised) Comprehension items. The items aren't widely available online, but cover topics such as 16 questions covering a wide range of problem situations involving such issues as health, social mores, social judgement, common sense, grasp of social conventions, interpersonal relations, and laws (e.g. “Why do people who want a divorce have to go to court?”). It should be noted that this component is but a small part of what makes up the overall IQ score.
While life might be easier for someone who knows what to do when confronted with a party room full of strangers, what in the Sam Hill does that have to do with collegiate academic abilities? Why should someone who can handle that situation (or can describe how they would handle it) be judged competent to take advanced English and calculus courses? Why is "common sense" being added to a test of cognitive abilities? What about when common sense and creativity are contradictory? (There are several things you could do to get a free lunch, but very few things you should do). And why should free-form creative writing be added when a test of writing skills is already scheduled for inclusion on the SAT in 2005?
Why add these squishy, controversial, subjective concepts to college admissions tests at all?
Why, to close the racial score gap, of course:
As for Sternberg, he says it's his job to invent a test that reflects a more nuanced view of intelligence - not, at this point, to sell that view to the American people. ''If we used the Rainbow Project, you'll find the kids at community colleges who do as well or better than the kids at Yale,'' he said...
...Sternberg...developed what he called the theory of ''successful intelligence,'' which holds that all people have three kinds of abilities: the analytical ability that is the focus of most standardized testing; the practical ability he also calls ''street smarts''; and the creative ability to adjust and invent. Underpinning all three is the ''tacit knowledge'' that helps people handle everyday encounters and situations. All but analytical ability, he says, are measures overlooked by traditional tests such as the SAT.
''There are people who are really good at traditional tests, who may get 800s, and then when they get out of school, that's the end of the story for them,'' Sternberg said. ''They don't get along with people. They don't persuade people to listen to them.''
Funny, I haven't read about many people who make perfect scores on the SAT but lead terrible lives. I haven't read about many brilliant people who consider their post-collegiate lives to be "the end of the story." The urban legends of the brilliant but antisocial geeks tend to be just that - legends.
Does Sternberg have much data to support his theories? There's his work with the Rainbow Project, in which he attempted to measure creative ability by giving a test to 1,000 subjects. The examinees were asked to dictate stories, write stories, and write captions for cartoons. Points were given for "cleverness, humor, and originality." He allegedly found smaller score gaps on this test than on the SAT, and better prediction of college performances, but I'd like more specifics than this article provides.
How were the students recruited? What was their baseline performance on academic measures? Could they make huge spelling and grammatical errors and still get high scores based on creativity? Which colleges did they ultimately attend? How much smaller were the score gaps on this test than on the SAT? How long did it take to train the raters, and how reliable were their ratings (reliability being a measure of generalizability)? Training a few raters to judge "cleverness, humor, and originality" for 1000 examinees is one thing; training raters who can do this consistently for the 1.5 million examinees who take the SAT every year is another.
What's more, if questions like "If Y = 5 and X = 6, what is Y+X?" are accused of being racially biased, how are you going to design a key for the item, "What would you do if you walked into a party where you didn't know anyone?" that is acceptable to everyone? I would argue that, for this item, there are indeed multiple correct answers based on culture - so many, in fact, that I'd be hard put to say what "correct" means in this case. Could that squishiness be the reason for the reduced score gap; i.e., a wide range of responses were considered to be "correct"?
Notice, too, that the BG reporter skips over asking the obvious question, which is that if an SAT score gap exists, why is the focus on creating new tests that avoid that gap, rather than modifying the K-12 education so that the score gap closes naturally? Why does it make more sense to start measuring "creativity" than to make sure that minority kids learn to read earlier? Why is the focus on the ends, and not the means?
Not everyone is buying Sternberg's theories:
...Sternberg's endeavor is fraught with uncertaintly, not least because his questions challenge traditional concepts of intelligence.
Camara said Sternberg's questions read ''almost like a personality test'' and ''will have to pass the smell test'' with parents and educators. College administrators balk at the idea of requiring another test, which could put them at a competitive disadvantage in the race for applicants. And parents might wonder why their children's future was riding on unorthodox measurements such as skill in writing captions for cartoons.
And parents would be right to do so. If their kids attend public schools, they've been exhaustively tested on reading and mathematics basics, not creativity. If they've been attending private schools, they've most likely been enrolled in demanding academic classes and haven't had much time to waste on writing cartoon captions.
...Despite the calls for a more comprehensive test, said the College Board's Camara, it has been difficult to persuade universities to contribute to the massive endeavor of researching a new test. Then, when it's ready, will admissions offices, parents, and high school principals put credence in a test that asks students how they would deal with an unwelcome guest?
''The biggest issue about viability is not whether we think it's viable, it's whether colleges do it,'' he said. ''It honestly takes courage, it takes good science and good results, and even after that it takes one or two institutions which have a lot of courage to say `We're going to use it.'''
Better get to work on that "good science" part of this, Mr. Camera.
The purpose of the SATs is to evaluate students' likelihood of success in college, not whether they'll do well in later life or be fun at parties. If judging the humor of their cartoon captions is a better predictor of college success than asking them to solve math problems, so be it. I can just see the SAT prep courses where students will drill in "tacit knowledge" and creativity.
Her points are good, especially the one about the creativity drills. If a test is high-stakes, then, regardless of its content, the creation of prep courses for it will be a lucrative business. The claim that a creativity- or common-sense-based test reduces the gap between rich and not-so-rich students will then become moot.
Posted by kswygert at August 11, 2003 01:28 PM