‘What would my peers say?’ Comparing the opinion-based method with the prediction-based method in Continuing Medical Education course evaluation
DOI:
https://doi.org/10.36834/cmej.77580Abstract
Background: Although medical courses are frequently evaluated via surveys with Likert scales ranging from “strongly agree” to “strongly disagree,” low response rates limit their utility. In undergraduate medical education, a new method with students predicting what their peers would say, required fewer respondents to obtain similar results. However, this prediction-based method lacks validation for continuing medical education (CME), which typically targets a more heterogeneous group than medical students.
Methods: In this study, 597 participants of a large CME course were randomly assigned to either express personal opinions on a five-point Likert scale (opinion-based method; n = 300) or to predict the percentage of their peers choosing each Likert scale option (prediction-based method; n = 297). For each question, we calculated the minimum numbers of respondents needed for stable average results using an iterative algorithm. We compared mean scores and the distribution of scores between both methods.
Results: The overall response rate was 47%. The prediction-based method required fewer respondents than the opinion-based method for similar average responses. Mean response scores were similar in both groups for most questions, but prediction-based outcomes resulted in fewer extreme responses (strongly agree/disagree).
Conclusions: We validated the prediction-based method in evaluating CME. We also provide practical considerations for applying this method.
Metrics
References
Etter JF, Perneger TV. Analysis of non-response bias in a mailed health survey. J Clin Epidemiol 1997;50:1123-1128. https://doi.org/10.1016/S0895-4356(97)00166-2 DOI: https://doi.org/10.1016/S0895-4356(97)00166-2
Asch DA, Jedrziewski MK, Christakis NA. Response rates to mail surveys published in medical journals. J Clin Epidemiol 1997;50:1129-1136. https://doi.org/10.1016/S0895-4356(97)00126-1 DOI: https://doi.org/10.1016/S0895-4356(97)00126-1
Johnson T, Owens L. Survey response rate reporting in the professional literature. Proceedings of the 58th Annual Meeting of the American Association for Public Opinion Research, Nashville; 2003.
Yu J, Cooper H. A quantitative review of research design effects on response rates to questionnaires. J Market Res. 1983:36-44. https://doi.org/10.1177/002224378302000105 DOI: https://doi.org/10.1177/002224378302000105
Spooren P, Brockx B, Mortelmans D. On the validity of student evaluation of teaching: The state of the art. Rev Educ Res. 2013;83:598-642. https://doi.org/10.3102/0034654313496870 DOI: https://doi.org/10.3102/0034654313496870
Rodriguez E, Marquett R, Hinton L, McBride M, Gallagher-Thompson D. The impact of education on care practices: an exploratory study of the influence of “action plans” on the behavior of health professionals. Intern psychogeriat. 2010;22:897-908. https://doi.org/10.1017/S1041610210001031 DOI: https://doi.org/10.1017/S1041610210001031
Skubleny D, Switzer N, Karmali S, de Gara C. Endoscopy services and training: a national survey of general surgeons. Can J Surg. 2015;58:330-334. https://doi.org/10.1503/cjs.015914 DOI: https://doi.org/10.1503/cjs.015914
Van Geest JB, Johnson TP, Welch VL. Methodologies for improving response rates in surveys of physicians: a systematic review. Eval Health Profess. 2007;30:303-321. https://doi.org/10.1177/0163278707307899 DOI: https://doi.org/10.1177/0163278707307899
Yarger JB, James TA, Ashikaga T, et al. Characteristics in response rates for surveys administered to surgery residents. Surgery 2013;154:38-45. https://doi.org/10.1016/j.surg.2013.04.060 DOI: https://doi.org/10.1016/j.surg.2013.04.060
Asch S, Connor SE, Hamilton EG, Fox SA. Problems in recruiting community-based physicians for health services research. J Gen Intern Med. 2000;15:591-599. https://doi.org/10.1046/j.1525-1497.2000.02329.x DOI: https://doi.org/10.1046/j.1525-1497.2000.02329.x
Brouns JW, Berkenbosch L, Ploemen-Suijker FD, Heyligers I, Busari JO. Medical residents perceptions of the need for management education in the postgraduate curriculum: a preliminary study. Int J Med Educ 2010;1:76-82. https://doi.org/10.5116/ijme.4cd8.43f1 DOI: https://doi.org/10.5116/ijme.4cd8.43f1
Hofstee W. Uitbuiting van de dagelijkse ervaring: Facetten van een psychometrische waarheidsbenadering [Exploitation of the daily experience]. Nederlands Tijdschrift voor de Psychologic 1993;48:277-287.
Hofstee W, Schaapman H. Bets beat polls: Averaged predictions of election outcomes. Acta Politica 1990;25:194-207.
Cohen-Schotanus J, Schönrock-Adema J, Schmidt HG. Quality of courses evaluated by ‘predictions’ rather than opinions: Fewer respondents needed for similar results. Med Teach. 2010;32:851-856. https://doi.org/10.3109/01421591003697465 DOI: https://doi.org/10.3109/01421591003697465
Schönrock‐Adema J, Lubarsky S, Chalk C, Steinert Y, Cohen‐Schotanus J. ‘What would my classmates say?’An international study of the prediction‐based method of course evaluation. Med Ed. 2013;47:453-462. https://doi.org/10.1111/medu.12126 DOI: https://doi.org/10.1111/medu.12126
Tomes T, Coetzee S, Schmulian A. Prediction-based student evaluations of teaching as an alternative to traditional opinion-based evaluations. Assess Eval Higher Educ. 2019:1-15. https://doi.org/10.1080/02602938.2019.1594157 DOI: https://doi.org/10.1080/02602938.2019.1594157
Bacchi S, Guo B, Brown S, Symonds I, Hudson JN. Can Australian medical students' predictions of peers' responses assist with gaining reliable results on course evaluations? Focus Health Profess Educ. 2018;19:14. https://doi.org/10.11157/fohpe.v19i2.250 DOI: https://doi.org/10.11157/fohpe.v19i2.250
Abrahams MB, Friedman CP. Preclinical course-evaluation methods at U.S. and Canadian medical schools. Acad Med 1996;71:371-374. https://doi.org/10.1097/00001888-199604000-00015. DOI: https://doi.org/10.1097/00001888-199604000-00015
Griffin P, Coates H, Mcinnis C, James R. The development of an extended course experience questionnaire. Qual Higher Educ. 2003;9:259-266. https://doi.org/10.1080/135383203200015111 DOI: https://doi.org/10.1080/135383203200015111
Norman G. Likert scales, levels of measurement and the "laws" of statistics. Adv Health Sci Educ Theory Pract 2010;15:625-632. https://doi.org/10.1007/s10459-010-9222-y DOI: https://doi.org/10.1007/s10459-010-9222-y
le Cessie S, Goeman JJ, Dekkers OM. Who is afraid of non-normal data? Choosing between parametric and non-parametric tests. Euro J Endocrinol. 2020;182:E1-E3. https://doi.org/10.1530/EJE-19-0922 DOI: https://doi.org/10.1530/EJE-19-0922
Altman DG, Bland JM. How to obtain the P value from a confidence interval. Bmj. 2011;343:d2304. https://doi.org/10.1136/bmj.d2090 DOI: https://doi.org/10.1136/bmj.d2304
Louangrath P. Minimum sample size method based on survey scales. Int J Res Methodol Soc Sci 2017;3:44-52.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Jamie S Chua, Merel van Diepen, Marjolijn D Trietsch, Friedo W Dekker, Johanna Schönrock-Adema, Jacqueline Bustraan
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Submission of an original manuscript to the Canadian Medical Education Journal will be taken to mean that it represents original work not previously published, that it is not being considered elsewhere for publication. If accepted for publication, it will be published online and it will not be published elsewhere in the same form, for commercial purposes, in any language, without the consent of the publisher.
Authors who publish in the Canadian Medical Education Journal agree to release their articles under the Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 Canada Licence. This licence allows anyone to copy and distribute the article for non-commercial purposes provided that appropriate attribution is given. For details of the rights an author grants users of their work, please see the licence summary and the full licence.
