The USA Today has finally noticed that, over the last couple of years, Yale University psychologist Robert Sternberg has been working on an alternative/augmentation to the SAT:
The test, dubbed "The Rainbow Project," evaluates creativity and problem-solving rather than analytical skills. Instead of multiple choice questions, it asks students to write captions for cartoons, outline how they would solve a problem, or write stories with unusual titles like "The Octopus's Sneakers" or "35,381."What most interests many experts about Sternberg's early experiments is that they appear to predict students' freshman GPA in college more accurately than SAT scores, and with a narrower gap between ethnic and socio-economic groups...
The test results could be interpreted as a threat to the College Board, which has funded Sternberg's research, and Sternberg says some in the testing field have reacted defensively. He is waiting to hear soon if the College Board will fund an expanded trial that would show if the patterns hold beyond the initial 800-student sample.
I've mentioned Sternberg on N2P before, and was in fact very skeptical of his items. I find it hard to believe that anyone could think this would ever replace the SAT.
This is not the first article published on Sternberg's work, by far, seeing as how this project has been going on for a while; this APA article from 2003 notes that Sternberg doesn't see his test as being a replacement for the SAT:
The test is based on a theoretical framework developed by Sternberg called the triarchic theory of successful intelligence. According to the theory, every person possesses a unique mix of analytical, creative and practical abilities. Although schools often only recognize and reward analytical skills, Sternberg said, creative and practical skills can be more useful in real-world settings--helping to spark innovation in the workplace, for example, or allowing Alaskan children to safely navigate the wilderness in conditions unsafe for the average person. Those that are successfully intelligent, said Sternberg, recognize and capitalize on their strengths in the three areas of intelligence and correct or compensate for their weaknesses."The traditional tests like the SAT and the ACT provide quite reasonable measures of analytical ability," he explained. "What we're trying to do is to supplement with additional analytical measures, but especially with creative and practical measures."
Those who disagree with Sternberg most likely think it's easier to guess/fake the answers to his exams:
For the time being, students know that their answers to quirky questions — like how to handle gossipy co-workers — don't matter. But if in the future the answers affect their college admissions, test subjects may be tempted to bluff — a problem that employer-administered personality tests have. "You can't fake solving a math problem," says Linda Gottfredson, a professor of education at the University of Delaware. "You can fake conscientiousness."
Or, like me, they just wonder if the score gap closes because there are so many right answers to each item. I also wonder how he's going to address the standard claims of bias that accompany every test nowadays. As I said on Aug 11, 2003:
What's more, if questions like "If Y = 5 and X = 6, what is Y+X?" are accused of being racially biased, how are you going to design a key for the item, "What would you do if you walked into a party where you didn't know anyone?" that is acceptable to everyone? I would argue that, for this item, there are indeed multiple correct answers based on culture - so many, in fact, that I'd be hard put to say what "correct" means in this case. Could that squishiness be the reason for the reduced score gap; i.e., a wide range of responses were considered to be "correct"?
Update: Psychometrician John Ray had this to say about Sternberg's work last year:
One of the people best known for disparaging the importance of IQ (general problem-solving ability) is Robert Sternberg. He acknowledges the reality of IQ but says it is only one of three types of important mental functioning that can be measured. But the other two he puts forward are pretty desperate proposals. The second one he himself summarizes as "street smarts" so is nothing more than knowledge of a particular culture or environment -- and nobody has ever disputed that you need both intelligence and knowledge to solve problems well. So while knowledge is important, to refer to it as an "ability" is evasive. And his third alternative is creativity -- which again seems reasonable at first. The problem with creativity, however, is that there appears to be no such thing...So Sternberg is still left with IQ as the single useful generalization about abilities.Posted by kswygert at March 5, 2005 02:15 PMAnd the criticisms of his work simply seem to have driven Sternberg to retreat further and further into dishonesty. He simply ignores whole bodies of data -- including things he had acknowledged in his own earlier work -- as Linda Gottfredson (PDF) points out.