Research like this should be replicated across medical schools heavy on the UKCAT

[krazyfish]

Does the UKCAT predict Year 1 performance in medical school?
Bonnie Lynch, Rhoda MacKenzie, Jon Dowell, Jennifer Cleland, Gordon Prescott
Medical Education, Vol. 43, No. 12. (December 2009), pp. 1203-1209

The need to identify the best applicants for medicine and to ensure that selection is fair and ethical has led to the development of alternative, or additional, selection tools. One such tool is the United Kingdom Clinical Aptitude Test, or UKCAT. To date there have been no studies of the predictive validity of the UKCAT.

This study set out to identify whether UKCAT total score and subtest scores predict Year 1 outcomes in medical school.

Year 1 students starting in 2007 at the University of Aberdeen or University of Dundee medical schools were included.

Data collected were: UKCAT scores; Universities and Colleges Admissions Service (UCAS) form scores; admission interview scores; final Year 1 degree examination scores, and records of re-sitting examinations and of withdrawing from a course. Correlations were used to select variables for multiple regression analysis to predict examination scores.

Data were available for 341 students. Examination scores did not correlate with UKCAT total or subtest scores. Neither UCAS form score nor admission interview score predicted outcomes. None of the UKCAT scores were reliably associated with withdrawals (P-values for all comparisons > 0.05). Only the decision analysis subtest was associated with re-sits of examinations, but the difference in means was contrary to the direction anticipated (P = 0.025, 95% confidence interval = 6.1201389.7).

UKCAT scores did not predict Year 1 performance at the two medical schools. Although early prediction is arguably not the primary aim of the UKCAT, there is some cause for concern that the test failed to show even the small-to-moderate predictive power demonstrated by similar admissions tools.


Thoughts anyone?

[nikusm]

Do the same research on BMAT unis and you'll see the difference.

[Professor Higgins]

It's not just the UKCAT, there are similar indications for the GAMSAT:

1) ACER, the publisher of GAMSAT and an organisation with a high level of respectability in the test publishing industry, conducted research in 2007 with 351 medical students which showed that scores on the GAMSAT aptitude test only predicted subsequent performance in medical school when used in conjunction with GPA (Grade Point Average) and interview scores (ACER/Coates, 2007) (4).

2) Groves et al. (2007) found no relationship between GAMSAT aptitude test scores and clinical reasoning and diagnostic thinking (significant aspects of medical practice) among 187 medical school students (5).

3) Wilkinson et al. (2008) in a study of 706 medical students in an Australian graduate‐entry medical program found that GPA (prior academic performance) was the strongest predictor of academic performance on the program and that GAMSAT scores were unrelated to academic performance (6).

That is, the only people who suggest that GAMSAT in any way indicates academic performance are ACER themselves.

http://www.psychometrik.com/Controve...%20Doctors.pdf

With the selection process being so abysmal, what think you of the rest of the educational process?

[krazyfish]

Do the same research on BMAT unis and you'll see the difference.

Well obviously there will be a difference. It is easy to tell that the BMAT offers far stronger construct validity, as an exam with better predictive ability. No test is perfect though, but certainly I believe that the BMAT, MCAT and GAMSAT are far better than the UKCAT.

[krazyfish]

It's not just the UKCAT, there are similar indications for the GAMSAT:



That is, the only people who suggest that GAMSAT in any way indicates academic performance are ACER themselves.

http://www.psychometrik.com/Controve...%20Doctors.pdf

With the selection process being so abysmal, what think you of the rest of the educational process?

Wow, interesting research. But I'm not surprised that researchers were able to find that GPA is the strongest ability. This is because GPA is more reflective of how one studies and prepares for courses and examinations. Why would any entrance exam be a better indicator?

But if there had to be a comparison or choice, wouldn't you all agree that the UKCAT is at the bottom, under all the other medical school exams?

[scohoust]

I'm as sceptical about the UKCAT as most of you are, but this paper really needs to be taken in context. The UKCAT has been proposed as a method to judge who will make good future doctors. It makes no prediction that these people need to perform particularly well in medical school. Indeed I'm not overly sure many people who do excellent academically in medical school (especially 1st year) necessarily make the better doctors.

As far as I'm aware, this research is ongoing towards looking at the 3rd year of this cohort - at least a good few months away! With 3rd year beginning to turn clinical, this will probably have a bit more weight.

Regardless - which impacts more on you lot - this isn't going to be quite enough to convince many Unis to get rid of the UKCAT altogether. Whether they'll alter how it's used - who knows.

Still, interesting reading.

[jubzy]

you know guys, i have spoken to university lecturers and professors, and they say that the sheer amount of work that the admissions tutors have to do to select applicants is overwhelming. The sheer volume of applications, A* grades, excellent PS (that were probably wrote by proefessionals) means that inevitably, there are always ratios of around 15:1 (exaggeration here, but u get what i mean).
So really, they have no choicce but to use a DEselection procedure, yes, i said deselection because in actual fact, that's what it is.
The other alternative to UKCAT is to just get all the applications that pass the minimum entry requirements, and then THROW THEM UP IN THEW AIR; the ones that the admissions tutors catch will be the lucky applications that make it through to med school. (im not joking, this is what my lecturer said).

Objection! Birmingham have been fine without selection tests for years. but even they are now "actively considering" the GAMSAT etc. So it seems that if the trends increase, and medicine sees record-high application numbers, then these deselection tests will be set in stone. theres no point arguing about them, the unis wont listen.

i dont know, im rambling now.

[Dick Cheney]

Traumatised, what you suggest is already happening. There is a natural trend for healthcare to become more systematic and protocol based. This systematising of healthcare facilitates roles previously performed by doctors to be performed by technicians. Some of the technicians have grandiose names such a paramedics and prescribing nurses but technicians they are. In not so many years (30?), doctors as we understand them will not exist.

[krazyfish]

you know guys, i have spoken to university lecturers and professors, and they say that the sheer amount of work that the admissions tutors have to do to select applicants is overwhelming. The sheer volume of applications, A* grades, excellent PS (that were probably wrote by proefessionals) means that inevitably, there are always ratios of around 15:1 (exaggeration here, but u get what i mean).
So really, they have no choicce but to use a DEselection procedure, yes, i said deselection because in actual fact, that's what it is.
The other alternative to UKCAT is to just get all the applications that pass the minimum entry requirements, and then THROW THEM UP IN THEW AIR; the ones that the admissions tutors catch will be the lucky applications that make it through to med school. (im not joking, this is what my lecturer said).

Objection! Birmingham have been fine without selection tests for years. but even they are now "actively considering" the GAMSAT etc. So it seems that if the trends increase, and medicine sees record-high application numbers, then these deselection tests will be set in stone. theres no point arguing about them, the unis wont listen.

i dont know, im rambling now.

Interesting anecdote, but seriously disturbing, about throwing the applications in the air. Not to sound blunt or direct or anything, but shouldn't the admissions already know what they are getting into? Receiving thousands of applications is overwhelming, yes, but they should know. Its like as if a traffic cop complaining that there he or she is overwhelmed by the amount of incoming cars that need to be told to go or stop. If they must use an alternative, you say, to the UKCAT, why not use a better more comprehensive test?

I know there is no point arguging, but I was merely pointing out the research that was published about the predictive value of the UKCAT. If enough of this research gets replicated, and similar results continue to suggest no correlation whatsoever, the universities will have to listen at some point. Being that this is the 3rd year (correct me if I'm wrong) that the UKCAT is introduced, it might still take a while. If more research show similar results, the universities will listen.

[krazyfish]

I'm as sceptical about the UKCAT as most of you are, but this paper really needs to be taken in context. The UKCAT has been proposed as a method to judge who will make good future doctors. It makes no prediction that these people need to perform particularly well in medical school. Indeed I'm not overly sure many people who do excellent academically in medical school (especially 1st year) necessarily make the better doctors.

As far as I'm aware, this research is ongoing towards looking at the 3rd year of this cohort - at least a good few months away! With 3rd year beginning to turn clinical, this will probably have a bit more weight.

Regardless - which impacts more on you lot - this isn't going to be quite enough to convince many Unis to get rid of the UKCAT altogether. Whether they'll alter how it's used - who knows.

Still, interesting reading.

On one hand, if the UKCAT does not make predictions on how people will do in medical school, then why use it? I'm sure the exam was designed to serve a similar purpose as the MCAT, LSAT, GMAT or GAMSAT as a useful indicator of one's potential in graduate school.

If the UKCAT was designed to see if people could become good future doctors, then it has some serious design flaws. Namely:

Verbal - Even blindly guessing, you have 1/3 of chance of getting it correct. Yeah, my point exactly. Name me another test designed to see if people will become good doctors that has this kind of design construct.

Quantitative - Math level is real easy. But testing condition? Not so. I doubt physicians would constantly need to make decisions in seconds (which is what you need to do if you actually want to attempt every single problem in the test), especially if you are in internal medicine where your patients come in by the appointment. Maybe this section is better suited to screen for candidates intended for the emergency room, where quick decisions are made all the time, more frequently than a clinic.

Abstract - Tests one ability to recognize patterns when encountering foreign objects. Sure, but it does not mean that if you can't see the pattern between # of right angles and circles, you can't recognize patterns in real life. Lacks real construct validity here.

Decision Analysis - Tests how well one can interpret different data or medical history when presented in different, unfamiliar formats. Similar to the abstract design flaw. If you can't strictly follow the verbal translations of the codes (including too many meanings or too less meanings), it doesn't mean one will be less adequate to read medical history. I would be more worried about not knowing enough, rather than what is presented. Chances are, you will get a good idea of what is presented, but if you know too less, its no use.

My point is that no entrance exam is perfect, and I can understand why universities want to use this test. Its not entirely bad, but its not the best either. A good decision is hard to make, given all the admission factors, but we can all conclude that the best decision should always be made.

Given that there are arguable flaws of the UKCAT, placing such a high emphasis on it (like a no wiggle room cut off) is not exactly what we can agree as the best decision for a test lacking in construct validity. It should be used in conjunction with everything else if they have to use it.