Unpacking SoTL “Failure”: Lessons from a Fellowship Program Investigating Generative AI’s Impacts
DOI:
https://doi.org/10.20343/teachlearninqu.14.39Keywords:
failure, educational development, generative AI use in the classroom, technology-enhanced learning, center for teaching and learningAbstract
The experience of failure in designing, implementing, interpreting, and/or disseminating Scholarship of Teaching and Learning (SoTL) research is not new to educational developers; it is, however, a relatively new focus for extended inquiry and exploration. The word failure may evoke complicated emotions or assumptions, but normalizing and learning from SoTL failure helps educational developers and instructors refine and elevate their SoTL practice. This paper contributes to the growing literature on SoTL failure by exploring project-level SoTL failure in the context of a fellowship program implemented by a Center for Teaching and Learning (CTL). This fellowship is designed to support novice SoTL researchers from all disciplines in evaluating the causal impact of technology-enhanced learning (TEL) interventions, specifically generative AI use, on student learning outcomes. Defined within the context of this fellowship, a failed project is one that did not collect at least one direct measure of student learning, did not analyze (or could not effectively interpret) the data regarding implications for future course design and delivery, or did not disseminate findings. We identify multiple sites for potential SoTL project failure within the processes of study design, implementation, and dissemination and consider how TEL interventions present further potential complications. Our goal is to share lessons learned through projects that failed or nearly failed as well as the preventative and mitigating strategies and scaffolding we developed to address the most common causes of failure.
Image Credit: Judy Brooks
References
Bastani, Hamsa, Osbert Bastani, Alp Sungu, Haosin Ge, Özge Kabakcı, and Rei Mariman. 2024. “Generative AI Can Harm Learning.” The Wharton School Research Paper. https://doi.org/10.2139/ssrn.4895486.
Bowen, Jose A. and Charles E. Watson. 2024. Teaching with AI: A Practical Guide to a New Era of Human Learning. Johns Hopkins University Press and American Association of Colleges and Universities.
Chick, Nancy L. 2014. “Methodologically Sound’ Under the ‘Big Tent’: An Ongoing Conversation.” International Journal for the Scholarship of Teaching and Learning 8 (2): 1. https://doi.org/10.20429/ijsotl.2014.080201.
Chick, Nancy L., Laura Cruz, Jennifer C. Friberg, and Hillary H. Steiner. 2023. “Making Space for Failure in the Scholarship of Teaching and Learning: A Blueprint.” Teaching & Learning Inquiry 11. https://doi.org/10.20343/teachlearninqu.11.36.
Colby, Susan A., Laura Cruz, Danielle Cordaro, and Clare Cruz. 2022. “Fellow Travelers: Taking Stock of Faculty Fellows Programs in the Age of Organizational Development.” To Improve the Academy: A Journal of Educational Development 41 (2): 2. https://doi.org/10.3998/tia.844.
Eberly Center. 2023. “Generative AI Teaching as Research (GAITAR) Initiative.” Effective October 3. https://www.cmu.edu/teaching/gaitar/index.html.
Eberly Center. 2025a. “GAITAR Fellows Project Descriptions.” Effective December 19. https://www.cmu.edu/teaching/gaitar/project-descriptions/index.html.
Eberly Center. 2025b. “Generative AI Tools FAQ.” Effective May 19. https://www.cmu.edu/teaching/technology/aitools/index.html.
Fan, Yizhou, Luzhen Tang, Huixiao Le, Kejie Shen, Shufang Tan, Yueying Zhao, Yuan Shen, Xinyu Li, and Dragan Gašević. 2025. “Beware of Metacognitive Laziness: Effects of Generative Artificial Intelligence on Learning Motivation, Processes, and Performance.” British Journal of Educational Technology 56 (2): 489–530. https://doi.org/10.1111/bjet.13544.
Healey, Mick, Kelly E. Matthews, and Alison Cook-Sather. 2019. “Writing Scholarship of Teaching and Learning Articles for Peer-Reviewed Journals.” Teaching & Learning Inquiry 7 (2). https://doi.org/10.20343/teachlearninqu.7.2.3.
Kruger, J. and D. Dunning. 1999. “Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments.” Journal of Personality and Social Psychology 77 (6): 1121.
Lovett, Marsha, Chad Hershock, and Judy Brooks. 2023. “Cultivating and Sustaining a Faculty Culture of Data-Informed Teaching: How Centers for Teaching and Learning Can Help.” In In Their Own Words: What Scholars and Teachers Want You to Know About Why and How to Apply the Science of Learning in your Academic Setting, edited by Catherine. E. Overson, Christopher M. Hakala, Lauren L. Kordonowy, and Victor A. Benassi. Society for the Teaching of Psychology.
Magrill, Jamie, and Barry Magrill. 2024. “Preparing Educators and Students at Higher Education Institutions for an AI-Driven World.” Teaching & Learning Inquiry 12: 1–9. https://doi.org/10.20343/teachlearninqu.12.16.
Marquis, Elizabeth, Mick Healey, and Michelle Vine. 2014. “Building Capacity for the Scholarship of Teaching and Learning (SoTL) Using International Collaborative Writing Groups.” International Journal for the Scholarship of Teaching and Learning 8 (1): 12. https://doi.org/10.20429/ijsotl.2014.080112.
Melville, Michael C., Chad Hershock, and B. Reeja-Jayan. 2024. “Virtual Engineering in Minecraft: Helping Students Visualize and Manipulate the Properties of Materials.” Advances in Engineering Education 13 (1): 11–26. https://doi.org//10.18260/3-1-1153-36059.
Mollick, Ethan. 2024. Co-Intelligence: Living and Working with AI. Penguin.
Moore, Jessie L., Peter Felten, and Michael Strickland. 2013. “Supporting a Culture of Writing: Faculty Writing Residencies as a WAC Initiative.” In Working with Faculty Writers, edited by Ann E. Geller and Michele. A. Eodice. Utah State University Press.
Noy, Shakked, and Whitney Zhang. 2023. “Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence.” Science 381 (6654): 187–92. https://doi.org/10.1126/science.adh258.
Pankiewicz, Maciej, and Ryan S. Baker. 2023. “Large Language Models (GPT) for automating feedback on programming assignments.” International Conference on Computers in Education. https://doi.org/10.58459/icce.2023.950.
Pardos, Zachary A., and Shreya Bhandari. 2024. “ChatGPT-Generated Help Produces Learning Gains Equivalent to Human Tutor-Authored Help on Mathematics Skills.” Plos One 19 (5). https://doi.org/10.1371/journal.pone.0304013.
Pincus, Jacqueline and Chad Hershock. 2024. “A Narrative Review of Interdisciplinary Teaching and Learning Scholarship: Closing the Gap Between Theory and Practice. International Journal for Academic Development 30 (4): 535–52. https://doi.org/10.1080/1360144X.2024.2389446.
POD Network. 2025. What is Educational Development. https://podnetwork.org/about/what-is-educational-development/.
Proffitt, Rachel, Anna E. Boone, William E. Janes, Jamie B. Hall, Samantha Shea Lemoins, and Winnie Dunn. 2023. “Supporting Faculty Scholarship through a Peer Writing Group: A Model and Guide for Success.” International Journal for Academic Development 30 (2): 201–15. https://doi.org/10.1080/1360144X.2023.2293864.
Skarupski, Kimberly A., and Kharma C. Foucher. 2018. “Writing Accountability Groups (WAGs): A Tool to Help Junior Faculty Members Build Sustainable Writing Habits.” Journal of Faculty Development 32 (3): 47–54.
Sun, Dan, Azzeddine Boudouaia, Chengcong Zhu, and Yan Li. 2024. “Would ChatGPT-Facilitated Programming Mode Impact College Students’ Programming Behaviors, Performances, and Perceptions? An Empirical Study.” International Journal of Educational Technology in Higher Education 21 (14). https://doi.org/10.1186/s41239-024-00446-5.
Wright, Mary C., Cynthia J. Finelli, Deborah Meizlish, and Inger Bergom. 2011. “Facilitating the Scholarship of Teaching and Learning at a Research University.” Change: The Magazine of Higher Learning 43 (2): 50–6. https://doi.org/10.1080/00091383.2011.550255.
Wright, Mary C., Deborah R. Lohe, Tershia Pinder-Grover, and Leslie Ortquist-Ahrens. 2018. “The Four Rs: Guiding CTLs with Responsiveness, Relationships, Resources, and Research.” To Improve the Academy 37 (2): 271–86. https://doi.org/10.1002/tia2.20084.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Chad Hershock, Laura Ochs Pottmeyer, Jacqueline Pincus, H. Elisabeth Ellington, Zach Mineroff
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
