Considering the potential unintended consequences of RateMDs: an exploratory study in one specialty
DOI:
https://doi.org/10.36834/cmej.77821Abstract
Background: Websites that facilitate communication between patients regarding their experiences with individual physicians are now relatively commonplace. Given patient-generated ratings are publicly available, physicians could use these to access rarely available patient feedback. We explored the content of reviews associated with low physician ratings and consider the potential benefits and consequences of relying on this form of freely available data to support individual life-long learning.
Methods: We conducted an exploratory qualitative descriptive study. We collected narrative comments associated with low numerical ratings on one physician-rating website (RateMDs) drawn from one specialty in Canada. Written reviews associated with low numerical ratings (≤2/5) for Canadian otolaryngologists were collected yielding a total of 878 comment sets that were analyzed deductively and iteratively.
Results: We found that patient comments described poor performance in areas that aligned, for the most part, with the CanMEDS roles including Professional, Communicator, and Leader; specifically referring to management of the clinical environment, administrative staff, and trainees.
Conclusion: While not intended for physician feedback, physicians could access patient-to-patient ratings and associated written reviews as a means to identify areas of practice improvement. However, this represents an unintended use of these websites. While speculative, access to patient-to-patient rating websites could negatively impact physician confidence or self-worth – representing a negative consequence of their use. The utilization of these data for potential self-improvement represents an unintended use of patient-to-patient ratings and so may be accompanied by unintended consequences for physicians who use these data as potential feedback, and patients who contribute to physician rating sites.
References
1. Boud D. Sustainable assessment: Rethinking assessment for the learning society. Stud Contin Educ. 2000; 22(2): p. 151-167. https://hdl.handle.net/10536/DRO/DU:30071986.
2. Gipps CV. Beyond testing: towards a theory of educational assessment. London. Washington: The Falmer Press; 1994. https://doi.org/10.4324/9780203486009.
3. Rushton A. Formative assessment: a key to deep learning? Med Teach. 2005; 27(6): p. 509-513. https://doi.org/10.1080/01421590500129159.
4. Schuwirth LWT, Van der Vleuten CPM. Programmatic assessment: from assessment of learning to assessment for learning. Med Teach. 2011; 33(6): p. 478-485. https://doi.org/10.3109/0142159X.2011.565828.
5. Shepard LA. The role of assessment in a learning culture. Educ Res. 2000; 29(7): p. 4-14. Https://doi.org/10.3102/0013189X029007004.
6. Nicol DJ, Macfarlane‐Dick D. Formative assessment and self‐regulated learning: a model and seven principles of good feedback practice. Stud High Educ. 2006; 31(2): p. 199-218. https://doi.org/10.1080/03075070600572090.
7. Tamblyn R, Abrahamowicz M, Brailowsky C et al. Association between licensing examination scores and resource use and quality of care in primary care practice. JAMA. 1998; 280(11): p. 989-996. https://doi.org/10.1001/jama.280.11.989
8. Tamblyn R, Abrahamowicz M, Dauphinee WD et al. Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA. 2007; 298(9): p. 993-1001. https://doi.org/10.1001/jama.298.9.993.
9. Tamblyn R, Abrahamowicz M, Dauphinee WD et al. Association between licensure examination scores and practice in primary care. JAMA. 2002; 288(23): p. 3019-3026. https://doi.org/10.1001/jama.288.23.3019.
10. Emmert M, Meier F, Pisch F, Sander U. Physician choice making and characteristics associated with Using physician-rating websites: cross-sectional study. J Med Internet Res. 2013; 15(8). https://doi.org/10.2196/jmir.2702.
11. Emmert M, Sander U, Pisch F. Eight questions about physician-rating websites: a systematic review. J Med Internet Res. 2013; 15(2). https://doi.org/10.2196/jmir.2360.
12. Hanauer DA, Zheng K, Singer DC, Gebremariam A, Davis MM. Public awareness, perception, and use of online physician rating sites. JAMA. 2014; 311(7): p. 734-735. https://doi.org/10.1001/jama.2013.283194.
13. Aungst H. Patients say the darnedest things. You can't stop online ratings, but you can stop fretting about them. Med Econ. 2008; 85(23): p. 27. PMID: 19209533.
14. Bacon N. Will doctor rating sites improve standards of care? Yes. BMJ. 2009; 338. https://doi.org/10.1136/bmj.b1030.
15. Greaves F, Ramirez-Cano D, Millet C, Darzi A, Donaldson L. Harnessing the cloud of patient experience: using social media to detect poor quality healthcare. BMJ Qual Saf. 2013; 22(3): p. 251-255. http://dx.doi.org/10.1136/bmjqs-2012-001527.
16. Lagu T, Hannon NS, Rothberg MB, Lindernauer PK. Patients’ Evaluations of health care providers in the era of social networking: an analysis of physician-rating websites. J Gen Intern Med. 2010; 25(9): p. 942-946. https://doi.org/10.1007/s11606-010-1383-0.
17. McCartney M. Will doctor rating sites improve the quality of care? No. BMJ. 2009; 338. https://doi.org/10.1136/bmj.b1033.
18. Pasternak A., Scherger JE. Online reviews of physicians: what are your patients posting about you? Fam Pract Manag. 2009; 16(3): p. 9. PMID: 19492765.
19. Hanauer DA, Zheng K, Singer DC, Gebremariam A, Davis MM. Parental awareness and use of online physician rating sites. Am Acad Pediatr. 2014; 134(4): p. 966-975. https://doi.org/10.1542/peds.2014-0681.
20. Strech D. Ethical principles for physician rating sites. J Med Internet Res. 2011; 13(4): p. 113. https://doi.org/10.2196/jmir.1899.
21. Trehan SK, Daluiski A. Online patient ratings: why they matter and what they mean. Hand Surg. 2016; 41(2): p. 316-319. https://doi.org/10.1016/j.jhsa.2015.04.018.
22. Kadry B, Chu LF, Gammas D, Marcario A. Analysis of 4999 online physician ratings indicates that most patients give physicians a favorable rating. J Med Internet Res. 2011; 13(4): p. 95. https://doi.org/10.2196/jmir.1960.
23. Sobin L, Goyal P. Trends of online ratings of otolaryngologists: what do your patients really think of you? JAMA Otolaryngol. Head Neck Surg. 2014; 140(7): p. 635-638. https://doi.org/10.1001/jamaoto.2014.818.
24. Yu J, Samuel LT, Yalçin S, Sultan AA, Kamath AF. Patient-recorded physician ratings: what can we learn from 11,527 online reviews of orthopedic surgeons? J Arthroplasty. 2020 Jun;35(6). https://doi.org/10.1016/j.arth.2019.11.021
25. Ramkumar PN, Navarro SM, Chughtai M, La T Jr, Fisch E, Mont MA. The patient experience: an analysis of orthopedic surgeon quality on physician-rating sites. J Arthroplasty. 2017 Sep;32(9):2905-2910. https://doi.org/10.1016/j.arth.2017.03.053
26. Akbarpour M, Tawk K, Frank M, Gomez AS, Mostaghni N, Abouzari M. Assessment of laryngologists' ratings on physician review websites. World J Otorhinolaryngol Head Neck Surg. 2023 Mar 31;10(1):1-6. https://doi.org/10.1002/wjo2.95
27. Chua JT, Nguyen E, Risbud A, et al. online ratings and perceptions of pediatric otolaryngologists. Laryngoscope. 2021 Oct;131(10):2356-2360. https://doi.org/10.1002/lary.29479
28. Calixto NE, Chiao W, Durr ML, Jiang N. Factors impacting online ratings for otolaryngologists. Ann Otol Rhinol Laryngol. 2018 Aug;127(8):521-526. https://doi.org/10.1177/0003489418778062
29. Stanbouly D, Rahhal Z, Talis A, Stanbouly R, Baron M, Arce K, Chandra SR. Assessing reviews of academic oral and maxillofacial surgeons within the US on Healthgrades. Oral Maxillofac Surg. 2024 Mar;28(1):323-330. https://doi.org/10.1007/s10006-023-01146-6
30. López A, Detz A, Ratanawongsa N, Sarkar U. What patients say about their doctors online: a qualitative content analysis. J Gen Intern Med. 2012; 27(6): p. 685-692. https://doi.org/10.1007/s11606-011-1958-4.
31. Ellimoottil C, Hart A, Greco K, Quek ML, Farooq A. Online reviews of 500 urologists. J. Urol. 2013; 189(6): p. 2269-2273. https://doi.org/10.1016/j.juro.2012.12.013.
32. Kinast RM, Barker GT, Day SH, Gardiner SK, Mansberger SL. Factors related to online patient satisfaction with ophthalmologists. J. Ophthalmol. 2014; 121(9): p. 1843-1845. e1. https://doi.org/10.1016/j.ophtha.2014.04.009.
33. Bakhsh W, Mesfin A. Online ratings of orthopedic surgeons: analysis of 2185 reviews. Am J Orthop. 2014; 43(8): p. 359-363. PMID: 25136868.
34. RateMDs. Available from www.RateMDs.com. [Accessed on Oct 15, 2015].
35. Stolberg M. Active euthanasia in pre-modern society, 1500–1800: learned debates and popular practices. Soc Hist Med. 2007; 20(2): p. 205-221. https://doi.org/10.1093/shm/hkm034.
36. Sandelowski M. What's in a name? Qualitative description revisited. Res Nurs Health. 2010; 33(1): p. 77-84. https://doi.org/10.1002/nur.20362.
37. Miles MB, Huberman AM, Saldana J. Qualitative data analysis: a methods sourcebook. Los Angeles: SAGE; 2013. https://www.jstor.org/stable/24332877.
38. Adams SA. Sourcing the crowd for health services improvement: the reflexive patient and “share-your-experience” websites. Soc Sci Med. 2011; 72(7): p. 1069-1076. https://doi.org/10.1016/j.socscimed.2011.02.001.
39. Canada, R.C.o.P.a.S.o. CanMEDS Framework. 2015 [Accessed on Aug 14, 2018].
40. Mayer B. The dynamics of conflict resolution: a practitioner's guide. New York: John Wiley & Sons; 2010.
41. Watling C, Driessen E, Vander der Vleuten CP, Lingard L. Learning from clinical work: the roles of learning cues and credibility judgements. Med Educ. 2012; 46(2): p. 192-200. https://doi.org/10.1111/j.1365-2923.2011.04126.x.
42. Krasner MS, Epstein RM, Beckman H et al. Association of an educational program in mindful communication with burnout, empathy, and attitudes among primary care physicians. JAMA. 2009; 302(12): p. 1284-1293. https://doi.org/10.1001/jama.2009.1384.
43. Siedsma M, Emlet L. Physician burnout: can we make a difference together? J Crit Care. 2015; 19(1): p. 273. https://doi.org/10.1186/s13054-015-0990-x.
44. Sood A, Prasad K, Schroeder D, Varkey P. Stress management and resilience training among Department of Medicine Faculty: a pilot randomized clinical trial. J Gen Intern Med. 2011; 26(8): p. 858-861. https://doi.org/10.1007/s11606-011-1640-x.
45. West CP, Dyrbye LN, Rabatin JT et al. Intervention to promote physician well-being, job satisfaction, and professionalism: a randomized clinical trial. JAMA Inten Med. 2014; 174(4): p. 527-533. https://doi.org/10.1001/jamainternmed.2013.14387.
46. Kluger AN, DeNisi A. The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin. 1996; 119(2): 254-284. https://doi.org/10.1037/0033-2909.119.2.254.
47. Lefroy J, Watling C, Teunissen PW, Brand P. Guidelines: the do’s, don’ts and don’t knows of feedback for clinical education. Perspect Med Educ. 2015; 4(6): p. 284-299. https://doi.org/10.1007/s40037-015-0231-7.
48. Sargeant J, Mann K, Sinclair D, Vleuten C, Metsemakers J. Understanding the influence of emotions and reflection upon multi-source feedback acceptance and use. Adv Health Sci Educ Theory Pract. 2008; 13(3): p. 275-288. https://doi.org/10.1007/s10459-006-9039-x.
49. Archer JC. State of the science in health professional education: effective feedback. Med Educ. 2010; 44(1): p. 101-108. https://doi.org/10.1111/j.1365-2923.2009.03546.x.
50. Samora JB, Lifchez SD, Blazar PE. Physician-rating web sites: ethical implications. J Hand Surg AM. 2016; 41(1): p. 104-110. e1. https://doi.org/10.1016/j.jhsa.2015.05.034.
51. Coulter A, Locock L, Ziebland S, Calabrease J. Collecting data on patient experience is not enough: they must be used to improve care. BMJ. 2014; 348. https://doi.org/https://doi.org/10.1136/bmj.g2225.
52. Coulter A. Can patients assess the quality of health care? Patients' surveys should ask about real experiences of medical care. BMJ. 2006; 333(7557): p. 1-2. https://doi.org/10.1136/bmj.333.7557.1.
53. Hodges B. Scylla or Charybdis: navigating between excessive examination and naive reliance on self‐assessment. Nurs. Inq. 2007; 14(3): p. 177-177. https://doi.org/10.1111/j.1440-1800.2007.00376.x.
54. Baldie DJ, Guthrie B, Entwistle V, Kroll T. Exploring the impact and use of patients’ feedback about their care experiences in general practice settings—a realist synthesis. Fam Pract. 2018. 35(1): p. 13-21. https://doi.org/10.1093/fampra/cmx067.
55. Gleeson H, Calderon A, Swami V, Deighton J, Wolpert M, Edbrooke-Childs J. Systematic review of approaches to using patient experience data for quality improvement in healthcare settings. BMJ Open. 2016; 6(8): p. e011907. https://doi.org/10.1136/bmjopen-2016-011907.
56. Wensing ME, Vingerhoets E, Grol R. Feedback based on patient evaluations: a tool for quality improvement? Patient Educ Couns. 2003; 51(2): p. 149-153. https://doi.org/10.1016/s0738-3991(02)00199-4.
57. Donnon T, Al Ansari A, Al Alawi S, Violato C. The reliability, validity, and feasibility of multisource feedback physician assessment: a systematic review. Acad Med. 2014; 89(3): p. 511-516. https://doi.org/10.1097/ACM.0000000000000147
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Kristina H Pulkki, Shamira Pira, Meredith Young, Grace M Scott, Carol Nhan, Kevin Fung, Gabriella Le Blanc, Lily HP Nguyen
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Submission of an original manuscript to the Canadian Medical Education Journal will be taken to mean that it represents original work not previously published, that it is not being considered elsewhere for publication. If accepted for publication, it will be published online and it will not be published elsewhere in the same form, for commercial purposes, in any language, without the consent of the publisher.
Authors who publish in the Canadian Medical Education Journal agree to release their articles under the Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 Canada Licence. This licence allows anyone to copy and distribute the article for non-commercial purposes provided that appropriate attribution is given. For details of the rights an author grants users of their work, please see the licence summary and the full licence.
