Evidence of the Impact of Scholarship of Teaching and Learning Purposes

This paper identifies a need for empirical studies to validate the purposes of the Scholarship of Teaching and Learning (SoTL) and reports the results of an investigation into one purpose based on one definition. The SoTL movement needs to be seen to be scholarly and to be engaging in evidencebased practice. More evidence is needed on whether SoTL is achieving its purposes and whether those purposes are worthwhile. The study was designed to address this issue by assessing whether one acknowledged purpose of SoTL (to enhance students’ experience of learning) is being achieved. Questionnaires on SoTL and approaches to teaching were used to capture the experiences of 56 faculty. It is concluded that when teaching is seen as scholarly and inquiring and when it is made pub lic and peer reviewed, there is a stronger likelihood that the improving student learning purposes of SoTL are being achieved.


INTRODUC TION
The focus of this paper is on two elements of the Scholarship of Teaching and Learning (SoTL) that I now see as fundamental to the continued growth of the idea of SoTL. The first concerns the debate about definitions of the Scholarship of Teaching and Learning and the sec ond (and main focus) is on the perceived purposes of SoTL.
It was widely acknowledged at the 2011 annual conference of the International Society for the Scholarship of Teaching and Learning (ISSOTL) that there is a range of ways of conceiving of the Scholarship of Teaching and Learning, that there is respect for such diversity, and that it is time to move beyond the search for a common definition or a unifying conceptualisation. However, the corollary of the acceptance of such diversity is a need for clarity in communicating about SoTL, because what is meant by SoTL is related to its intended purpose, and how the value of that purpose is perceived.
As with the definition, there can be multiple purposes of SoTL. The purpose that seems to be most frequently cited is a way to enhance university teaching. Among the many others described is a way to raise the status of teaching, a means through which teachers may come to teach more knowledgeably, a means through which the quality of teaching may be assessed, a way to enhance students' experience of learning, a means by which the research profile of an in di vidual or department may be enhanced, and a way of stimulating interest in teaching. Other purposes are apparent in a list of "SoTL Areas of Impact" described by Ciccone (2008). These include a means of strengthening development programs for higher education professionals, and a means of informing changes in institutional policies and procedures (Ciccone, 2008, p.13).
For the SoTL movement to be taken seriously, there needs to be a scholarly evidencebased practice associated with these purposes. In addition to the many papers on what SoTL is, and the many reported examples of good practice, there is a need for studies that provide some evidence that however SoTL is being practiced, the proposed purposes are worthwhile and are being achieved. Chalmers (2011) recently reviewed the studies that looked at whether some or all forms of SoTL have assisted in raising the status of teaching and concluded: "The evidence suggests that on measures of perception, promotion, tenure and salary, teaching continues to be relegated to a lower status with promotion, tenure and higher salary directed to those who do disciplinary research and publish" (Chalmers, 2011, p.32). Of the other purposes, much remains unknown. For example, evidence is needed on whether SoTL has enabled better assessment of the quality of teaching, or whether SoTL as research is a worthwhile pursuit.
It is not my aim in this paper to argue for a single definition of SoTL or for a single purpose. As noted above, one of the more significant developments in the growing SoTL movement is the acceptance that there are many ways of defining SoTL and that such diversity is both welcomed and accepted. However, the dangers of not treating such diversity in a scholarly way became very apparent to me in 2012, when I attended a conference that included a "provocation" session. The topic was related to whether all faculty should be expected to be involved with SoTL, and three speakers each argued for a position that they decided would be provocative. In all three cases, no definitional position was articulated and in two cases several of the definitions of SoTL were conflated. For example, one speaker at one stage talked as though all SoTL was educational research, proceeded to criticise much of it for being poor quality research, and concluded that SoTL should not be an expectation of all teachers. That SoTL should be an expectation of all teachers is a statement derived from one definition (that SoTL is an integral part of teaching), and it is more akin to reflection than to research. The provocation, for me at least, was a lack of a definition of the SoTL that was being criticized, and in not acknowledging that some of what was being criticised may have been generated to meet an alternate purpose using a different definition. Differing definitions may all lead to reasonable statements, but they may have quite different meanings and consequences. The danger in not setting the definitional context was seen in the reaction by one participant, new to teaching, who left the session quite confused about SoTL and its value, and fearful of what it was that their university required of them.
In this paper, I define SoTL as described elsewhere (Trigwell & Shale, 2004): We see scholarship as being about making scholarly processes transparent and publicly available for peer scrutiny.… We use Andresen's (2000) ideas to describe a scholarly process as involving personal, but rigorous, intellectual development, inquiry and action built on values such as honesty, integrity, open-mindedness, scepticism and intellectual humility. We see teaching as a scholarly process aimed at making learning possible (Ramsden, 1992). It therefore follows that we see the scholarship of teaching as about making transparent, for pub lic scrutiny, how learning has been made possible (Trigwell, et al., 2000)…. This way of conceptualising the notion of a scholarship of teaching, therefore, is activity orientated. We share with the Carnegie Academy for the Scholarship of Teaching and Learning, the view that scholarship of teaching is fundamentally an aspect of the activity of teaching. The focus is on teaching as an act, but with an outcome derived from scholarly inquiry and practice. Shulman describes how an activity can relate to scholarship: 'For an activity to be designated as scholarship it should manifest at least three key characteristics: It should be public, susceptible to criti cal review and evaluation, and accessible for exchange and use by other members of one's scholarly community… A scholarship of teaching will entail a public account of some or all of the full act of teaching-vision, design, enactment, outcomes, and analy sis-in a manner susceptible to criti cal review by the teacher's professional peers and amenable to productive employment in future work by members of that same community.' (Shulman, 1998, pp. 5-6). (Trigwell & Shale, 2004, pp. 525-6) For the purpose of this paper, this description is both a definition and a hypothesis. There is very little empirical evidence that the teachers who engage with SoTL in this way are also improving their students' learning (Healey, 2000;Haigh, Gossman & Jiao, 2011), or even teaching in a way that is associated with higher quality learning, yet this is clearly the purpose of such activity. Nor is there evidence of the reverse of these relations: that teachers who adopt, say, one approach to teaching rather than another, in a particular context, are more likely to conceive of SoTL in one way rather than another. In short, the hypothesis is that there are systematic relations between conceptions of SoTL and the adoption of teaching approaches known to be more supportive of high quality student learning. Without the confirmation of this hypothesis, activity associated with this part of the SoTL movement is based on a set of beliefs, even though those beliefs are consistent with the results of respected research into what constitutes good teaching.

THE STUDY
The study described in this paper was designed to find out whether the teachers who experience SoTL in ways that are consistent with the definition above are also likely to be the teachers who describe an approach to teaching that is associated with students' deep approaches to learning and therefore to higher quality learning outcomes. A number of research studies have now been published on the relations between teachers' approaches to teaching and students' approaches to learning, and on students' approaches to learning and the quality of their learning outcome. These relations, described in more detail below, are taken as a given in this study. What remains as an unknown is the nature of the relations between approaches to teaching and how teachers engage with the scholarship of teaching.
In learning environments that encourage and assess student understanding (rather than recall), the students achieving the higher quality learning outcomes are found, on average, to be adopting the characteristics of a deep approach to learning, such as a search for meaning, relevance, application, and relations to other experience. When students adopt more of a surface approach (characterised by a focus on parts rather than wholes, and on recall in order to pass assessments) they achieve, on average, a lower-quality learning outcome (Lizzio, Wilson & Simons, 2002;Marton, Hounsell & Entwistle, 1997;Trigwell, Ashwin & Millan, 2012).
In separate studies, teachers' approaches to teaching have been found to be related to their students' (deep/surface) approaches to learning. An approach to teaching in which teachers provoke discussion and debate, monitor students' changing understanding, and encourage students to question their own ideas (a Conceptual Change / Student-Focused [CCSF] approach) has been shown, in several studies, to be related to more deep approaches to learning being adopted by their students (Trigwell, Prosser & Waterhouse, 1999;Gibbs & Coffey, 2004). The same studies show that when teachers report that they focus on, for example, on good presentation, covering the content, and providing a good set of notes (an Information Transfer / Teacher-Focused [ITTF] approach) their students are more likely to report adopting more of a surface or lower-quality approach to learning.
The question being addressed in this study extends this sequence of relations by asking whether those teachers reporting higher CCSF approaches to teaching are also the teachers who are more engaged with the Scholarship of Teaching and Learning (as defined above). If the answer is yes, then given the research showing relations between teaching and outcomes of learning, the scholarly, inquiring, peer review, and shared aspects of their teaching are likely to be achieving the purpose of improving student learning. If this is the case, this purpose of SoTL would be seen to have validity.

THE QUESTIONNAIRES
Questionnaires capturing two aspects of teachers' experience were needed in order to conduct this study. The first, an Approaches to Teaching Inventory (ATI), had already been developed (Trigwell, Prosser & Ginns, 2005). The sec ond, on the Scholarship of Teaching and Learning, had not been developed, and a brief description of the development process is provided below.
The ATI (Trigwell & Prosser, 2004;Trigwell, et al., 2005) was used to capture the self-reported approach to teaching adopted by teachers in the same context as their experienced of SoTL. The ATI contains 22 items. Eleven items form a Conceptual Change/ Student-Focused approach to teaching scale, in clud ing items such as "A lot of teaching time in this subject should be used to question students' ideas." Another 11 items form an Information Transmission/Teacher-Focused approach to teaching scale that focuses more on what the teacher does. For example, "My teaching in this subject focuses on delivering what I know to the students." The scholarship of teaching inventory needed to be able to gather experiences of key aspects of SoTL when it is defined as the process of making transparent, for peer review, how student learning is being made possible. Existing questionnaires do not have such a focus. For example the Scholarship of Teaching Questionnaire (STQ) (Kreber, 2001) was designed to measure scholars' and experts' conceptualisation of the scholarship of teaching in order to address confusing differences and generalise common patterns.
A Scholarship of Teaching Inventory (STI) was developed for this pilot study in two phases. First, the research literature and existing SoTL questionnaires were used to identify possible items in six broad dimensions consistent with SoTL as defined in this paper. The sources included the general SoTL literature (e.g. Hutchings & Huber, 2008;Weimer, 2006), the research results of previous explanatory data (e.g. Trigwell, et al., 2000), modified STQ items (Kreber, 2001), and items adapted from a Scholarship of Teaching and Learning Impact Survey (University of Wisconsin, 2009). The dimensions were linked to the definition as follows: a teachers' awareness of relevant pedagogic literature and a teaching-learning model (or concept, theory or framework) is needed in order to articulate how learning might be made possible. Inquiry and reflection are needed to ascertain whether student learning is being made possible, and peer review involves both collegiality and a pub lic process. The six SoTL dimensions were labelled use of literature, articulation of a teaching/learning model, inquiry, reflection, teaching as a pub lic activity, and peer review.
In the sec ond phase, a 37-item inventory was distributed, with the ATI, to 120 faculty in one university, and the 56 responses (47%) were used to select a subset of suitable items. The version of the STI resulting from this analy sis contains 12 items, three each in four SoTL dimensions: peer review, use of theory, teaching as a pub lic activity, and inquiry. In the initial analy sis, the reflection items did not form a coherent scale independent of the other scales, so those items were omitted. The items in the literature and model dimensions consistently loaded on the same factor and were combined to form one (theory) scale. Responses were collected on a 5-point Likert scale, from "strongly disagree" (value=1) to "strongly agree" (value=5). Defining items for each scale are as follows: Theory: I cannot explain what concepts or theories underpin my teaching in this subject.
(reverse scored) Public: I oft en make a deliberate effort to share my teaching experience with others.
Review: Improving my teaching is more effective if reviewed by my colleagues.
Inquire: I oft en investigate questions related to how students learn in my discipline.
The methods used and numerical results obtained in the analyses for this study are provided separately as an Appendix. The factor analy sis loadings and scale characteristics of the STI are given in Tables 1 and 2 of the Appendix.

Approaches to Teaching and the Scholarship of Teaching and Learning
The Scholarship of Teaching and Learning can be defined in many ways. In this paper, it is defined as having a primary focus on improving the learning of the teachers' students, while satisfying several key elements of scholarship: a scholarly inquiry leading to the production of a pub lic artefact and the peer review of that outcome. This is in contrast to a definition that sees scholarly journal publications as the major outcome, with a primary focus on faculty career development and contributions to new knowledge that may (or may not) lead to improved teaching and learning generally. If this locally-focused definition is a good definition, it follows that thinking and practicing in this way should be consistent with teaching that facilitates higher quality student learning. If a purpose of SoTL is to improve student learning, there should be a positive relationship between a more Conceptual Change/Student-Focused approach to teaching and the key elements of SoTL.
The results from this study do show a positive relationship between a more Conceptual Change/Student-Focused approach to teaching and the key elements of SoTL, and therefore validate the SoTL purpose of improving student learning. They show that when teachers describe a higher level of agreement with the dimensions of the Scholarship of Teaching and Learning as defined in this paper, they are more likely to report that they are using an approach to teaching that is more CCSF oriented and less ITTF oriented. High CCSF scores are associated with higher scores for use of theory (Pearson correlation co-efficient, R=.30), making teaching pub lic (R=.42), peer review (R=.43), and inquiry (R=.43) (Appendix, Table 3). Higher ITTF scores are associated with lower scores on use of theory (R= −.27) and peer review (R=−.36) ( Table 3).
The cluster analy sis (Appendix , Table 4) shows that there is a systematic set of relations between approaches to teaching and dimensions of SoTL. The 23 teachers who on average have higher scores on a CCSF approach to teaching than the remainder of the sample also have higher scores (on average) on all four SoTL dimensions. These other 33 teachers have lower CCSF scores and lower mean scores on all four SoTL dimensions (on average). Small differences between the groups on ITTF scores suggest that it is a greater awareness of CCSF approaches to teaching, rather than less of an awareness of ITTF approaches, that is associated with the Scholarship of Teaching and Learning.
As noted earlier, previous research suggests that higher CCSF approaches to teaching are more likely to be associated with higher-quality learning outcomes. Neither of the analytical methods used here give any indication of causality, in either direction (from conceptions of SoTL to approaches to teaching, or from teaching approach to SoTL engagement). What they do show is that teachers who are more aware of the value of the dimensions of SoTL used in this study are also the teachers likely to report adopting CCSF approaches to teaching. These results, in confirming the validity of this purpose of SoTL, also strengthen the argument that all teachers should be engaged in these forms of SoTL. This is the first report of a study linking the ways teachers experience these aspects of SoTL with the ways they approach their teaching. Relations between engagement with some forms of SoTL and student learning experience have been reported. Brew and Ginns (2008) showed that when teachers contribute more to a high departmental score on a scholarship index (calculated from points awarded for teaching qualifications, awards, and publications), their students also report experiencing a supportive learning context.

The Impact of Other Purposes of the Scholarship of Teaching and Learning
While these results confirm the validity of the student learning-focused definition of SoTL used here, they do not suggest that other definitions of SoTL are not also good definitions. Nor does this paper directly address any of the other purposes of SoTL. What the results do reveal is that empirical studies can be used to validate purposes of SoTL, and that there are gaps in this research area that need to be filled. Examples of questions that might be addressed are (a) What forms of SoTL are useful as a means of assessing the quality of teaching? (b) Is SoTL, when defined as research, making a worthwhile contribution? (c) In what ways is SoTL contributing to teachers becoming more knowledgeable about teaching, and are any of them achieving that purpose? Each of these three areas is considered briefly in the following three paragraphs.
At least two purposes can be seen in using SoTL in assessing the quality of teaching. First, if assessment drives practice, and if some teaching practices that include SoTL lead to better student learning, then incorporating SoTL into teacher assessment is desir-able. Second, if the purpose is to assess the quality of teaching, indicators of quality that supplement indicators associated with the act of teaching should also be included in the assessment process. These could include the SoTL dimensions of inquiry, literature use, and peer review. Chalmers (2011) describes the relations between SoTL and assessment of teaching in promotion. Another example is the Sydney University promotion process, which includes SoTL as one of five criteria in the academic promotion process (the other four are performance, leadership in teaching, research-led teaching, and student-focused teaching) (University of Sydney, 2012). Olsson and colleagues have also used SoTL to "distinguish and successfully examine levels of complexity in pedagogical reasoning and levels of theory based scholarship in pedagogical action" (Roxå, Olsson & Mårtensson, 2008, p. 288). But in all these cases, the validity of purpose (as a means of assessing the quality of teaching) has not been tested. The question "Is the quality of teaching higher when SoTL is used in the assessment process?" remains unanswered.
Is SoTL as research a worthwhile purpose? Teachers doing research into their own teaching and learning context has been one response to Boyer's idea of four scholarships of the professoriate (Boyer, 1990). It is difficult to see how this research approach can contribute to raising the status of teaching, relative to research. Whether or not it leads to better teaching or learning is not answerable from the research in this paper, but according to Hattie and Marsh (Hattie & Marsh, 1996;Marsh & Hattie, 2002) the correlation between teaching quality (assessed mainly using student ratings of teaching) and research (assessed mainly using grants and publications) is near zero. There is no evidence that the correlations are different in university education departments where research into teaching and learning is likely to be more widespread. However, in their study of research-teaching relations, Prosser and colleagues found that there is variation among researchers in their experience of research, and when research thinking is focused on the wider scenario, in clud ing the whole research field (rather than the parts or elements of projects or specific topics), there is a higher likelihood that in their teaching they will also describe using more of a CCSF approach (Prosser, Martin, Trigwell, Ramsden & Middleton, 2008). A similar variation in research experience might be expected among those involved in research into teaching and learning. High-quality research conducted from this perspective may therefore raise the status of teaching, but this also remains an unexplored research area.
If teachers are more knowledgeable about teaching, they might, on average, be expected to teach better and to facilitate better student learning. The teachers in this study who ascribe benefits to their teaching from the use of the pedagogic literature and who are able to articulate a model/framework/theory behind their teaching (theory scale) are likely to be more knowledgeable about teaching than those who do not. Of the four SoTL scales, the theory scale is correlated least strongly with the CCSF approach to teaching scale but the relations are positive (R= .30) and statistically significant. With the ITTF scale, the theory scale correlation is small and negative (R= -.27) but also statistically significant (p<.05). The outcomes of SoTL as research may also contribute to teacher knowledge (as described in the previous paragraph), but while increasing the knowledge of the teacher might be achieved using either of these forms of SoTL, the question still remains as to whether it is the increase in knowledge that leads to better teaching (and student learning).

CONCLUSION
This paper reports the results of an investigation into one purpose of SoTL (to enhance students' experience of learning) based on one definition. The study was designed to assess whether that purpose is being achieved, and the results suggest that it is. The teachers who adopt scholarly, inquiring, reflecting, peer reviewing, student-centred approaches to teaching are likely to be achieving the purpose of improving student learning, and the purpose therefore has validity. In this context, there is congruence between SoTL and teaching, and student learning is likely to be the beneficiary. This is a valid and achievable purpose of SoTL, but it highlights the need for information on the validity of other purposes of SoTL.

Keith Trigwell is Professor of Higher Education in the Institute for Teaching and Learning at the
University of Sydney.

APPENDIX: ANALYSIS ME THODS AND NUMERICAL RESULTS
Following tests of item skewness and a series of iterations of factor and scale reliability analyses, the 37 trial scholarship of teaching inventory items were reduced to 13 items ( Table 1). The item Public11 (I usually participate in conferences, meetings or courses about teaching and learning) did not load significantly on any one factor and was omitted from the final analyses. The four factors were used to construct four STI scales. The scale means and standards deviations, along with scale reliabilities (Cronbach alpha) are given in Table 2. The results of the study of the relations between teaching approach and dimensions of SoTL are presented here in two forms: as correlations between pairs of the six scales in Table 3, and as a cluster analy sis by in di vidual faculty members in Table 4.  A cluster analy sis groups faculty by maximising the similarity with others in the same group and by maximising the differences between the groups. It differs from the correlations shown in Table 3 by involving all variables in the same calculation. The hierarchical cluster analy sis (Ward's) method used for this sample shows that Group A, containing 23 faculty, has a higher mean score on the CCSF approach to teaching scale than Group B, and also has higher scores on all four SoTL variables than group B. All the differences are statistically significant at p<.01. The two groups show smaller differences in mean scores on the ITTF approach to teaching scale, and in this case the differences are not statistically significant.