Evaluation Utility Metrics (EUMs) in Reflective Practice

Authors

  • Ralph Renger Just Evaluation Services (JESS), LLC
  • Jessica Renger Claremont Graduate University
  • Richard N Van Eck University of North Dakota
  • Marc D. Basson University of North Dakota
  • Jirina Renger Walden University

DOI:

https://doi.org/10.3138/cjpe.72386

Keywords:

evaluation, utility, metrics, reflection, recommendations, evaluation recommendations, reflective practice

Abstract

The article proposes three evaluation utility metrics to assist evaluators in evaluating the quality of their evaluation. Afer an overview of refective practice in evaluation, the diferent ways in which evaluators can hold themselves accountable are discussed. It is argued that refective practice requires evaluators to go beyond evaluation quality (i.e., technical quality and methodological rigor) when assessing evaluation practice to include an evaluation of evaluation utility (i.e., specifcactions taken in response to evaluation recommendations). Three Evaluation Utility Metrics (EUMs) are proposed to evaluate utility: whether recommendations are considered (EUMc ), adopted (EUMa ), and (if adopted) the level of infuence of recommendations (EUM li ). The authors then refect on their experience in using the EUMs, noting the importance of managing expectations through negotiation to ensure that EUM data are collected and the need to consider contextual nuances (e.g., adoption and infuence of recommendations are infuenced by multiple factors beyond the control of the evaluators). Recommendations for increasing EUM rates by paying attention to the frequency and timing of recommendations are also shared. Results of implementing these EUMs in a real-world evaluation provide evidence of their potential value: practice tips led to an EUMc of 100% and and EUM a of over 80%. Methods for considering and applying all three EUMs together to facilitate practice improvement are also discussed.

Author Biographies

Ralph Renger, Just Evaluation Services (JESS), LLC

Dr. Renger was born in Calgary, Canada.   He spent 23 years teaching public health evaluation at the University of Arizona and the University of North Dakota before beginning his own consulting firm.  He is internationally known for his advancement of system evaluation theory and several applied program evaluation methodologies including the ATM approach for logic modeling, developing program theory using source documentation, and root cause analysis to facilitate process evaluation.

Marc D. Basson, University of North Dakota

Marc Basson received his MD from the University of Michigan, his PhD from Yale, and an MBA from Wayne State University.  A practicing general surgeon, he also serves as the Principal Investigator and Director of the DaCCoTA Center for Translational Research, an NIH-funded multi-institutional consortium dedicated to fostering clinical and translational biomedical research across North and South Dakota.  He is Professor of Surgery, Biomedical Sciences, and Pathology and Senior Associate Dean for Medicine and Research at the University of North Dakota School of Medicine and Health Sciences.

Jirina Renger, Walden University

Dr. Jirina Renger is a graduate from the University of Arizona, College of Public Health. She currently serves as a core faculty member in the DrPH and Ph.D. Public Health programs in the College of Health Professions at Walden University.  Her area of specialization is theory-driven evaluation and program theory reconstruction.

Downloads

Published

2022-04-25 — Updated on 2022-06-30

Issue

Section

Practice Note- English