The development of two Point of Care Ultrasound stations for Objective Structured Clinical Examinations in undergraduate medical education
DOI:
https://doi.org/10.36834/cmej.81229Abstract
Introduction: Point-of-care ultrasound (POCUS) is a valuable clinical skill that improves clinical care but requires substantial training. Validated assessment tools provide empirical evidence regarding trainee performance while also informing program-level evaluation. We developed two POCUS-specific stations for objective structured clinical examinations (OSCEs) to assess skill acquisition and inform best practices in undergraduate medical education.
Methods: A multidisciplinary group of POCUS educators identified two POCUS applications (pleural effusion and abdominal free fluid) well suited for the undergraduate level. A modified Delphi approach was used to develop POCUS-application-specific skill checklists and global rating scale. Two medical programs piloted the stations to inform reliability.
Results: Across two sites, 46 and 41 students participated in the pleural effusion and abdominal free fluid stations respectively. Checklists showed high internal reliability, with Cronbach’s alpha of 0.85 (95% CI 0.71-0.93) for the pleural effusion station and 0.87 (95% CI 0.74-0.95) for the abdominal free fluid station. Krippendorff’s alpha, a measure of inter-rater reliability, was also equally strong at 0.85 (95% CI 0.43-0.94) and 0.83 (95% CI 0.50-0.94) respectively.
Conclusion: Both POCUS OSCE stations demonstrated good internal and inter-rater reliability. Deployment of these OSCE stations at programs with integrated POCUS curricula may help refine programming and training expectations.
Downloads
References
1. Diaz-Gomez JL, Mayo PH, Koenig SJ. Point-of-care ultrasonography. N Engl J Med. 2021;385:1593-1602. https://doi.org/10.1056/nejmra1916062 DOI: https://doi.org/10.1056/NEJMra1916062
2. Sena A, Alerhand S, Lamba S. Milestone approach to designing a point-of-care ultrasound curriculum for transition-to-residency programs in the United States. Teach Learn Med. 2020;33(3):270-81. https://doi.org/10.1080/10401334.2020.1814296 DOI: https://doi.org/10.1080/10401334.2020.1814296
3. Tarique U, Tang B, Singh M, Kulasegaram KM, Ailon J. Ultrasound curricula in undergraduate medical education: a scoping review. J Ultrasound Med. 2018;37(1):69-82. https://doi.org/10.1002/jum.14333 DOI: https://doi.org/10.1002/jum.14333
4. Davis JJ, Wessner CE, Potts J, Au AK, Pohl CA, Fields JM. Ultrasonography in undergraduate medical education: a systematic review. J Ultrasound Med. 2018;37(11):2667-79. https://doi.org/10.1002/jum.14628 DOI: https://doi.org/10.1002/jum.14628
5. Steinmetz P, Dobrescu O, Oleskevich S, Lewis J. Bedside ultrasound education in Canadian medical schools: a national survey. Can Med Educ J. 2016;7(1):e78-e86. https://doi.org/10.36834/cmej.36646 DOI: https://doi.org/10.36834/cmej.36646
6. Russell FM, Zakeri B, Herbert A, et al. The state of point-of-care ultrasound training in undergraduate medical education: findings from a national survey. Acad Med. 2022;97(5):723-7. https://doi.org/10.1097/ACM.0000000000004512 DOI: https://doi.org/10.1097/ACM.0000000000004512
7. Hoppmann RA, Mladenovic J, Melniker L, et al. International consensus conference recommendations on ultrasound education for undergraduate medical students. Ultrasound J. 2022;14:31. https://doi.org/10.1186/s13089-022-00279-1 DOI: https://doi.org/10.1186/s13089-022-00279-1
8. Ma IWY, Steinmetz P, Weerdenburg K, et al. The Canadian medical student ultrasound curriculum: a statement from the Canadian Ultrasound Consensus for Undergraduate Medical Education Group. J Ultrasound Med. 2020;39(7):1279-87. https://doi.org/10.1002/jum.15218 DOI: https://doi.org/10.1002/jum.15218
9. Dinh VA, Lakoff D, Hess J, et al. Medical student core clinical ultrasound milestones: a consensus among directors in the United States. J Ultrasound Med. 2016;35(2):421-34. https://doi.org/10.7863/ultra.15.07080 DOI: https://doi.org/10.7863/ultra.15.07080
10. DeBiasio C, Pageau P, Shefrin A, Woo MY, Cheung WJ. Point-of-care-ultrasound in undergraduate medical education: a scoping review of assessment methods. Ultrasound J. 2023;15(1):30. https://doi.org/10.1186/s13089-023-00325-6 DOI: https://doi.org/10.1186/s13089-023-00325-6
11. Khan KZ, Ramachandran S, Gaunt K, Pushkar P. The objective structured clinical examination (OSCE): AMEE guide no. 81. Part I: an historical and theoretical perspective. Med Teach. 2013;35(9):e1437-e46. https://doi.org/10.3109/0142159x.2013.818634 DOI: https://doi.org/10.3109/0142159X.2013.818634
12. Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9):S63-S7. https://doi.org/10.1097/00001888-199009000-00045 DOI: https://doi.org/10.1097/00001888-199009000-00045
13. Patrício MF, Julião M, Fareleira F, Carneiro AV. Is the OSCE a feasible tool to assess competencies in undergraduate medical education? Med Teach. 2013;35(6):503-14. https://doi.org/10.3109/0142159x.2013.774330 DOI: https://doi.org/10.3109/0142159X.2013.774330
14. Sheppard G, Williams KL, Metcalfe B, et al. Using Kane’s framework to build an assessment tool for undergraduate medical student’s clinical competency with point of care ultrasound. BMC Med Educ. 2023;23:43. https://doi.org/10.1186/s12909-023-04030-9 DOI: https://doi.org/10.1186/s12909-023-04030-9
15. Damewood SC, Leo M, Bailitz J, et al. Tools for measuring clinical ultrasound competency: recommendations from the Ultrasound Competency Work Group. AEM Educ Train. 2019;4(Suppl 1):S106-S12. https://doi.org/10.1002/aet2.10368 DOI: https://doi.org/10.1002/aet2.10368
16. Royal College of Physicians and Surgeons of Canada. Objectives of training in the specialty of emergency medicine. Available at: https://www.royalcollege.ca/rcsite/documents/ibd/emergency_otr_e.pdf. [Accessed Dec 10, 2024].
17. Royal College of Physicians and Surgeons of Canada. Entrustable Professional Activities for anesthesiology Version 3.0. Available from: https://www.royalcollege.ca/rcsite/documents/cbd/epa-guide-anesthesiology-v3-e.pdf. [Accessed Dec 10, 2024].
18. Emergency ultrasound guidelines. Ann Emerg Med. 2009;53:550-70. https://doi.org/10.1016/j.annemergmed.2008.12.013 DOI: https://doi.org/10.1016/j.annemergmed.2008.12.013
19. Blehar DJ, Barton B, Gaspari RJ. Learning curves in emergency ultrasound education. Acad Emerg Med. 2015;22(5):574-82. https://doi.org/10.1111/acem.12653 DOI: https://doi.org/10.1111/acem.12653
20. Association of American Medical Colleges. Core EPAs guiding principles. Core Entrustable Professional Activities for Entering Residency Curriculum Developers’ Guide. Available at: https://store.aamc.org/downloadable/download/sample/sample_id/63/%20. [Accessed Dec 10, 2024].
21. Association of Faculties of Medicine of Canada. AFMC Entrustable Professional Activities for the Transition from Medical School to Residency. Available at: https://www.afmc.ca/wp-content/uploads/2022/10/AFMC_Entrustable-Professional-Activities_EN_Final.pdf. [Accessed Nov 30, 2024].
22. Steinmetz P, Oleskevich S, Dyachenko A, McCusker J, Lewis J. Accuracy of medical students in detecting pleural effusion using lung ultrasound as an adjunct to the physical examination. J Ultrasound Med. 2018;37:2545-52. https://doi.org/10.1002/jum.14612 DOI: https://doi.org/10.1002/jum.14612
23. Olszynski P, Russell M, Neufeld A, Malin G. The clinical ultrasonography elective in clerkship (CUSEC): a pilot elective for senior clerkship students at the University of Saskatchewan. Can Med Educ J. 2020;11(1):e144-6. https://doi.org/10.36834/cmej.61810 DOI: https://doi.org/10.36834/cmej.61810
24. Kern DE, Thomas PA, Hughes MT. Curriculum development for medical education: a six-step approach. Baltimore: Johns Hopkins University Press; 2009.
25. Bahner DP, Hughes D, Royall NA. I-AIM: a novel model for teaching and performing focused sonography. J Ultrasound Med. 2012;31(2):295-300. https://doi.org/10.7863/jum.2012.31.2.295 DOI: https://doi.org/10.7863/jum.2012.31.2.295
26. Hejri SM, Jalili M, Muijtjens AM, Van Der Vleuten CP. Assessing the reliability of the borderline regression method as a standard setting procedure for objective structured clinical examination. J Res Med Sci. 2013;18(10):887-91.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Ryan Good, Juliana Wilson, Joshua C Kaine, Vijay J Daniels, Janeve Dessy, Joshua Lloyd, Gillian Sheppard, Linden Kolbenson, Irene WY Ma, Arthur Au, Paul Adam Olszynski
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Submission of an original manuscript to the Canadian Medical Education Journal will be taken to mean that it represents original work not previously published, that it is not being considered elsewhere for publication. If accepted for publication, it will be published online and it will not be published elsewhere in the same form, for commercial purposes, in any language, without the consent of the publisher.
Authors who publish in the Canadian Medical Education Journal agree to release their articles under the Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 Canada Licence. This licence allows anyone to copy and distribute the article for non-commercial purposes provided that appropriate attribution is given. For details of the rights an author grants users of their work, please see the licence summary and the full licence.
