The advent of computer-based testing is worrisome to would-be med students:
As if pre-meds did not already have enough to worry about, recent changes to the MCAT have some pre-med students worried about more than just mastering its content.The Association of American Medical Colleges announced recently in a press release that it will convert the MCAT to a computer-based format within the next two years, a move that will force both students and test-prep companies to sharpen their strategies for tackling the test, rather than their pencils.
The paper format of the test will be administered through 2006, though trial versions of the computer-based test will be given at the August 2006 testing date.
There are many advantages to computer-based tests; three of the biggest ones are the shorter test length (I'm assuming here the new MCAT is adaptive as well as computerized), more testing opportunities, and shorter score report turnaround time. Based on my moderate experience with CBTs, I think that some of the fears of future examinees will turn out to be unfounded:
"With the computer-based test you can't underline passages and put notes next to text, so it's hard to map out the progression of a passage. You can't keep track of it when you have to keep switching from a computer to your notes," said Patrick Wiita, a fourth-year microbiology, immunology and molecular genetics student who has taken the MCAT.
Underlining text onscreen is a fairly simple feature to add, if it isn't already included. And it's also possible to allow a typing area on-screen for notes or comments, for those who don't feel comfortable using scratch paper.
There's also concern about computer glitches, Wiita said. "Who knows what errors could occur in programming. It's the same reason they haven't switched to online voting. If there's an error, there's no paper trail," Wiita said.
Why on earth would you want a paper trail when you can capture every keystroke? Certainly, big programming errors can occur - but paper tests can vanish in transit just as easily. If anything, the software that processes computerized testing errors tend to provide more information to the testing companies than paper errors. There will be exact records of when screens go down, where examinees were when the error occurred - and most CBT providers that I've seen have little problem restarting a test on the correct screen after an error occurs. Any testing company thinking about CBT administrations correctly places a lot of focus on error prevention and recovery.
Mustafa said new testing formats create a lot of work for test-prep companies, especially research into how students feel about the test. "We did a survey on 4,000 (students) to see how they're feeling, what they think. Eighty-two percent said they would do worse on a computer-based test," Mustafa said.
I've yet to see any research showing that examinees in fact do worse as a whole on computer-based tests. Some subgroups, in fact, do better in certain construct areas. Regardless, this is an easy enough question to answer; matched examinee groups can be compared with respect to P&P and CBT scores.
Of course, examinees may experience a feeling of doing worse if they're switching from a P&P test to an computer-adaptive test (or CAT), since the items will be tailored to their ability level and they'll see more of what they consider to be hard items. But CAT scoring scheme takes item difficulty into account, so that two examinees could miss the same number of items but end up with different scores, based on the difficulty of items that were answered correctly.
CATs have been operational in a high-stakes, large-scale environment since 1993 (when ETS pioneered the GRE-CAT), so there's quite a bit of theoretical and operational research out there to guide the development and refinement of the MCAT-CAT. Good luck to them.
Posted by kswygert at October 11, 2005 11:08 AM