Online Calibration for OSCE Examiners
N.B. The information below is authored by the mini-project applicants, not by staff of the subject centre. This text represents the views and opinions of the mini-project team only, not those of the subject centre or its affiliates.
Principal investigator
Dr Gerry Gormley , Queen's University Belfast
Full list of project partners
Dr Kieran McGlade, Clare Thomson
Topic
Clinical simulation, Quality enhancementBackground
What do we plan to do?
We plan to develop a prototype e-learning package that will provide examiners with a resource to enhance their skills in awarding global scores in Objective Structured Clinical Examinations (OSCEs). The proposed training package will present examiners with a series of video clips of candidates performing a variety of OSCE stations. Examiners will be required to award theoretical global scores and provide free text comments explaining their reasons. Results will be stored allowing examiners to compare their scores with others who have previously used the training package. Such feedback will allow examiners to reflect on their judgments and help calibrate their decisions for future OSCEs. On completion of the training package, examiners will be able to print off a certificate to indicate that they have taken part in the training exercise for both quality assurance and appraisal purposes.
What is the need for this planned project?
OSCEs have gained wide popularity in the assessment of clinical competence in healthcare education. In a controlled environment, candidates are directly observed performing various clinical tasks. Their performance is tested using a structured ‘checklist’. A passing standard is required to determine whether a student has demonstrated competence in each clinical task. In the UK the ‘modified borderline regression method’ has become the forerunner in standard setting OSCEs. In this method examiners are asked to consider a candidate’s overall performance and award a global score. This global score is then mathematically regressed against the ‘checklist’ score for each station – which then determines the passing mark. Crucial to this process is the training of examiners in awarding such global scores. Examiner training improves the reliability of OSCEs. The General Medical Council in their document ‘Tomorrow’s Doctors’ have highlighted the need for examiners to receive training to allow them to carry out their role consistently.
Why do we want to do it?
Awarding global scores is essentially a subjective process. Our experience is that there is a strong demand from examiners for further training in this area and they would value the opportunity to compare their scores with other examiners and discuss reasons behind differences in scores.
Why is it worth the funding?
There has never been a greater need for medical schools to professionalise and quality assure their assessment processes. Key to this is the training of examiners and setting credible passing standards – particularly in high stake examinations such as final MB. In this project we aim to enhance the current training of our examiners. Given the widespread use of OSCEs in healthcare education the potential applicability and scalability of this project is great.
Proposed activities
1. Project Management Group convenes to oversee delivery of project (Jan 2010).
2. Suitable OSCE stations selected and checklists developed (Jan 2010).
3. Logistics and story boards worked out for each OSCE station (Feb - March 2010).
4. Film the 12 OSCE stations. Patients will be played by actors from the Queen’s University ‘Patients as Partners’ initiative. The candidates will be played by volunteering students (April 2010).
5. Production of background learning and supporting materials – how to use the package, principles of conducting OSCEs, role of an OSCE examiner (March – April 2010).
6. Creation of package. The 12 videos will be added to a database and the functionality of the rating and feedback system implemented (Feb – May 2010).
7. Population of feedback/ratings – a minimum of ten experienced members of staff will rate and give qualitative feedback on each video in order to provide data for the first examiner to observe (June 2010).
8. Testing and implementation (Sept 2010).
9. A mixed method evaluation is planned (a) quantitative analysis of scores (b) qualitative analysis of comments with thematic analysis and (c) overall user satisfaction. (Sept-Jan 2011).
Proposed outcomes
Materials
-The main deliverable of this project is a prototype e-learning package for OSCE examiners in awarding global scores. Following a general introduction about OSCEs and awarding global scores, users will be presented with three randomly selected video clips appropriate to their level of examining (i.e. junior level vs. senior level OSCEs). These videos will portray simulated candidates performing a range of OSCE stations (e.g. communication, clinical assessment and procedural skills). Intentionally some of these skills well be performed well and others not (to allow examiners to observe a range of competencies). Examiners will then award a global score and comment on why they awarded this score. Each response will be compared to others who have previously used the package.
-A paper for publication in a peer-reviewed medical education journal
-A presentation for a peer-reviewed medical education conference
Benefits
This package aims to improve OSCE examiners skills in awarding global scores and reduce inter-rater variability. Furthermore given the challenges of releasing clinical examiners from their NHS activities - this training package allows examiners to use it in their own time. We would hope it will be of use to other institutions who also could contribute to its further development.
Expertise of grant holder and project team
Similar work
We are not aware of any other similar online training packages for OSCE examiners in awarding OSCE global marks.
References:
• Newble D. Techniques for measuring clinical competence: objective structured clinical examinations. Medical Education. 2004; 38:199-203.
• Harden RM, Gleeson FA. Assessment of clinical competence using an Objective Structured Clinical Examination (OSCE). Medical Education. 1979; 13:41-54.
• Schuwirth LWT, van der Vleuten CPM. The use of clinical simulation in assessment. Medical Education. 2003; 37:65-71.
• General Medical Council. Tomorrow’s doctors: Recommendations on Undergraduate Medical Education. London: GMC 2009.
Contact details
Grant holder: g.gormley@qub.ac.uk,
Queen's University Belfast
Amount awarded: £4170