Evaluation Utility Metrics (EUMs) in Reflective Practice
Keywords:evaluation, utility, metrics, reflection, recommendations, evaluation recommendations, reflective practice
The article proposes three evaluation utility metrics to assist evaluators in evaluating the quality of their evaluation. Afer an overview of refective practice in evaluation, the diferent ways in which evaluators can hold themselves accountable are discussed. It is argued that refective practice requires evaluators to go beyond evaluation quality (i.e., technical quality and methodological rigor) when assessing evaluation practice to include an evaluation of evaluation utility (i.e., specifcactions taken in response to evaluation recommendations). Three Evaluation Utility Metrics (EUMs) are proposed to evaluate utility: whether recommendations are considered (EUMc ), adopted (EUMa ), and (if adopted) the level of infuence of recommendations (EUM li ). The authors then refect on their experience in using the EUMs, noting the importance of managing expectations through negotiation to ensure that EUM data are collected and the need to consider contextual nuances (e.g., adoption and infuence of recommendations are infuenced by multiple factors beyond the control of the evaluators). Recommendations for increasing EUM rates by paying attention to the frequency and timing of recommendations are also shared. Results of implementing these EUMs in a real-world evaluation provide evidence of their potential value: practice tips led to an EUMc of 100% and and EUM a of over 80%. Methods for considering and applying all three EUMs together to facilitate practice improvement are also discussed.
The Canadian Evaluation Society (CES) owns the copyright to all material published in the Journal. Authors are informed of this policy prior to submission of the final copy for publication. Requests for permission to reprint, post or distribute copies of articles (electronic or hardcopy) come to the Editor-in-chief, who has the authority to decide on behalf of CES.