Comparing and Contrasting a Program versus System Approach to Evaluation: The Example of a Cardiac Care System


  • Ralph Renger Just Evaluation Services, LLC
  • Jirina Foltysova Walden University
  • Jessica Renger Claremont Graduate University
  • Stewart I. Donaldson Claremont Graduate University
  • Gary Hart University of North Dakota
  • Andrew Hawkins ARTD Consultants



systems, evaluation, systems thinking


This paper focuses on the application of systems thinking concepts to evaluate systems.  The terms systems and systems thinking concepts are first defined.  The use of systems thinking concepts in program and system evaluation are then highlighted.  It is noted that while there are methods available to assist evaluation practitioners in applying systems thinking concepts to program evaluation, there is a need for similar guidance in applying systems thinking concepts to evaluating systems.  System Evaluation Theory (SET) is then reviewed as one evaluation theory designed to apply systems thinking concepts to evaluating systems. A case illustration is presented to help teach practitioners how to apply SET’s three steps.  The discussion focuses on comparing the differences between evaluation questions answered by applying systems thinking concepts versus those using program logic models.

Author Biographies

Ralph Renger, Just Evaluation Services, LLC

Dr. Renger is president of Just Evaluation Services, LLC. He is internationally known for his advancement of system evealuation theory and several applied program evaluation methodologies including the ATM approach for logic modeling, developing program theory using source documentation, and root cause analysis to facilitate process evaluation.

Dr. Renger taught program and system evaluation at the University of Arizona and University of North Dakota for 23 years.  He is now the president of Just Evaluation Services, LLC.  He has published over 50 applied articles in the program and system domains to help evaluators transition from theory to practice so that their evaluations produce more meaningful and useful results.


Contact Information

Ralph Renger

Just Evaluation Services, LLC

14777, E. Circle M Ranch Place

Vail, AZ  85641




Jirina Foltysova, Walden University

Jirina Renger, DrPH

Dr. Jirina Renger is a graduate from University of Arizona, College of Public Health. She currently serves as a core faculty member in the DrPH and PhD Public Health programs in the College of Health Sciences at Walden University.  Her area of specialization is theory-driven evaluation and program theory reconstruction.

Contact Information

Jirina Renger

14777, E. Circle M Ranch Place

Vail, AZ  85641


Jessica Renger, Claremont Graduate University

Jessica Renger is a current graduate student in the Evaluation and Applied Research Methods PhD program at Claremont Graduate University. Before coming to Claremont, she received a bachelor’s degree in Neuroscience from the University of Arizona. She has prior experience assisting with health-related evaluation projects in Arizona, North Dakota, and Wyoming. She plans to pursue a career as both an evaluation practitioner and researcher in the future. 

Contact Information

Jessica Renger

Just Evaluation Services, LLC

14777, E. Circle M Ranch Place

Vail, AZ  85641




Stewart I. Donaldson, Claremont Graduate University

Stewart I. Donaldson, PhD  (note to Editor/publisher.  Please use Dr. Donaldson’s middle initial when listing the authors on the article cover page)

Stewart I. Donaldson, Ph.D. is Distinguished University Professor, Executive Director of the Claremont 

Evaluation Center, and Director of The Evaluators' Institute (TEI).  He has a large evaluation practice, 

publishes books and articles, and teaches a variety of courses on evaluation theory and science, and 

was President of the American Evaluation Association during the International Year of Evaluation in 2013. 


Contact Info


Dr. Stewart I. Donaldson

Distinguished University Professor

Executive Director, Claremont Evaluation Center (CEC)

Director, The Evaluators' Institute (TEI)

Claremont Graduate University

123 E. 8th Street Claremont, CA 91711

(909) 702-7316 (cell)

More info:

Gary Hart, University of North Dakota

Gary Hart, PhD

Dr. Hart is an expert in rural health research with an emphasis in health workforce and program evaluation. He is the director of one of the nations, largest, oldest, and most prestigious rural centers.

Contact Information

Gary Hart, PhD

Director & Professor

Center for Rural Health

School of Medicine & Health Sciences

Suite E231 (Office #252)

University of North Dakota

1301 North Columbia Road, Stop #9037

Grand Forks, ND     58202-9037


Office Phone:   (701) 777-3899

Fax:                  (701) 777-6779


Andrew Hawkins, ARTD Consultants


Andrew Hawkins is a partner ARTD Consultants, one of Australia’s most established specialist evaluation firms. He has a keen interest in translating evaluation theory into practice to generate evidence and insight for decision makers.


Contact Information


Andrew Hawkins

Level 4, 352 Kent St

Sydney NSW 2000








Practice Note- English