PEIG Strategy

---

1. What is the Program Evaluation and Improvement (PEI) Strategy?

1.1. Introduction

The PEI Strategy is a key and integral component of the Medicine Program at UNSW. Its overall purpose is –
Such evaluation and improvement processes and practices:

(a) ensure systematic and evidence-based continual improvement in key aspects of the Program (e.g. the student experience);
(b) provide effective support mechanisms to foster excellence and continual development in the Program (e.g. staff development and recognition programs); and
(c) enable clear communication with and between key stakeholders (e.g. by reporting) about the quality of the Program.

The high level guidance provided by the Strategy includes the articulation of general priorities, principles and values to shape the development of the more formal evaluation and improvement policies, processes and practices that are integrated with the Medicine Program.

This document (Part 1) presents the Overall Strategy. Part 2 describes the Strategy in Practice. To remain evolutionary and dynamic, the Strategy in Practice is updated on a continuing basis. A major result of the implementation of the PEI Strategy is the strengthening of the culture of critical reflection and improvement in the Faculty.

1.2. The Faculty’s leading-edge aspiration

The Faculty aspires to have the Medicine Program recognised for excellence, both nationally and internationally. The PEI Strategy is a critical factor and vehicle for achieving this vision.

To attain such “leading-edge” recognition over time, the PEI Strategy is:

(a) aligning itself with emerging good practice ideas in the literature on program quality assurance in higher education and medical education;
(b) innovative in its concepts and methods; and
(c) dynamic and adaptive over time.

1.3. The essence of the PEI Strategy: Fundamental priorities and values

Though more sophisticated in its detail, the PEI Strategy can be captured in brief by a small number of strategic principles. Each of these is a fundamental priority or value. Collectively these guide the development of evaluation and improvement processes and practices.

These strategic planks are discussed in more detail in section 3 below.

2. Why has the Faculty developed the PEI Strategy?

The Faculty wants to ensure that it has in place a systematic and explicit approach to ensuring high quality in the Medicine Program. A number of related reasons underpinned this need.

2.1. The Faculty’s fundamental concern for excellence

The Faculty of Medicine continually aspires to excellence in both its teaching and research. This goal is grounded in the Faculty’s fundamental concern for high quality in all that it does, and clearly this concern extends to its flagship Medicine Program (MB BS). Because the MB BS program is based on innovative design features, and relies on significantly different learning and teaching practices, the achievement of excellence depends on having in place effective and ongoing program evaluation, support and improvement processes.

2.2. The Faculty’s demonstration of quality and performance standards

Apart from the Faculty’s fundamental concern for high quality in the Medicine Program, there are a number of important external bodies and stakeholders that require the Faculty to demonstrate quality and performance standards.

The Australian Medical Council (AMC), which accredited the Medicine Program in 2003, is a critically important stakeholder for the Faculty. It has the ultimate authority to withdraw accreditation for the Program if the Council’s standards are not satisfied. The AMC’s standards cover many related aspects of the Program, including its effective evaluation. The Faculty’s case for accreditation presented to the AMC included its goal to develop a leading-edge approach to program evaluation and improvement.

In 2005, the Australian Universities Quality Agency (AUQA) conducted its first Quality Audit of UNSW. While the scope of this audit was for the whole of the University, the processes that exist at Faculty level to evaluate and improve quality were audited. Through the development of its Program Evaluation and Improvement Strategy, Medicine has provided UNSW with a potential exemplary process that it can showcase.

In addition to the expectations outlined above, UNSW and its Faculties are required to report a range of information concerning educational quality to the Federal Government. The PEI Strategy and the processes and information that generates both the Faculty’s and the University’s capacity to satisfy these requirements and position themselves effectively to obtain additional competitive funding to support quality development.

2.3. Attraction and standing

The Faculty aims to attract and retain the best students and staff, and continually enhance its international standing in medical education. To be able to do this into the future it will be necessary to demonstrate explicitly that quality is taken very seriously. To ensure that community and stakeholder perceptions of the Faculty remain high or are enhanced, it will be important to be able to communicate clear information about the quality of the student experience and their learning outcomes, and the capabilities of UNSW Medicine graduates. It will also be important to be able to demonstrate that the right kinds of supports and incentives are provided to enable staff and students to do excellent work.

3. The PEI Strategy: More details on priorities and its conceptual base

In section 1.3 the essence of the PEI Strategy was presented in overview form using six strategic principles. Each of these principles is explained below. Given the intended high level (overview) nature of this document, great detail is not provided. More detail can be seen in the evaluation, support and improvement practices developed under the guidance of the Strategy. It should be noted that these strategic principles overlap and should be considered as an integrated whole.

3.1. Attraction and standing

The choice of the words “Evaluation and Improvement” (for the Strategy) provides an important insight to its conceptual basis. The inclusion of and emphasis on “Improvement” reflects the importance of processes and actions that lead to change, as a result of evaluation.

The importance of action, following evaluation, seems obvious. However, the international literature (e.g. Harvey, 2003) and case evidence on evaluation and quality assurance in higher education (including reports by audit or accreditation agencies) point regularly to the failure of institutions to “close loops”. In practice this usually means that while there may be significant evaluation processes in place, systematic or explicit processes to facilitate improvement and/or communicate with relevant stakeholders are missing or inadequate. The PEI Strategy ensures that explicit and systematic processes exist to apply evaluation findings and processes to actual improvement, and that effective communication with stakeholders occurs.

In addition, approaches to program quality assurance that claim to be based on good practice need to include effective support mechanisms. Such mechanisms can include programs for continuing career or professional development for staff, incentive and recognition processes, and student support services.

3.2. An holistic view of the Program and quality

Fundamentally, the PEI Strategy is concerned with program quality and improvement. It is important to make clear what is meant by “program” and what perspective on “quality” is being adopted.

In order to help organise and implement effective and integrated evaluation, improvement and support, the Program as a whole has been conceived of as encompassing four broad and related Program Quality Aspects (PQAs). These are:

• Curriculum and Resources;
• Student Experience;
• Staff and Teaching; and
• Student and Graduate Outcomes

“Quality” (in the Program) is also conceived of in terms of these four PQAs, and the following perspectives and values (and see Figure 1 below).

• The primary values and goals of the Program provide the foundation for the concept of quality (e.g. the eight Graduate Capabilities; the commitment to students developing a lifelong approach to learning).

• Program quality relates principally to outcomes, processes and environments.

• The PQAs and their associated indicators provide information to facilitate evaluation and improvement of program outcomes, processes and experiences, and environments.

• Levels of quality, as requirements or aspirations, are viewed as being dynamic and evolving, and not in terms of fixed or absolute standards.

• Excellence is the aspiration and driver, and continual critical reflection, improvement and support are the key enablers.

• Program stakeholders help to define the criteria by which quality is understood and judged, and they also provide observations and judgements about quality (in relation to the different PQAs).

3.3. A primary concern for student and staff experiences and support

The PEI Strategy places a high priority on the experiences of and support provided for students and staff. Students and staff are primary stakeholders in the Program, and Program quality is in major part understood and determined by reference to the student experience (e.g. motivation and satisfaction levels), support provided to them, and the commitment and capabilities of Faculty staff. The evaluation and continual improvement of student and staff experiences, and the provision of appropriate support programs (e.g. student counselling, staff recognition and incentive) are important enablers of Program quality.

3.4. An emphasis on continual reflection and improvement based on appropriate quality indicators and evidence

The four Program Quality Aspects discussed in section 3.2 are broad in their concept and scope. Each aspect has a number of components. These components require indicators that provide more practical descriptors of the aspects/components. For example, two components of the aspect “Student Experience” might be ‘classroom experience’ and ‘experience of administrative services’. To be useful for program evaluation and improvement, practical quality indicators are needed for each of these components. For example, ‘student satisfaction with classroom teaching’ is one widely used indicator of the (component) ‘quality of classroom experience’.

3.5. Effective communication between key Program stakeholders

Effective communication between the main stakeholders of the Medicine Program is a critical success factor for the PEI Strategy. Obtaining and reporting the best possible information about program aspects, facilitating action for change, and developing the most effective program support mechanisms, all depend on effective two-way communication between stakeholders (e.g. students, staff, the AMC).

The PEI Strategy not only depends on effective communication. It also enables the improvement of communication within the Faculty and externally, particularly in relation to matters concerning program quality.

3.6. Integration of systematic evaluation, improvement and support processes

The principle and practice of integration are important in two senses. First, to be optimally effective processes and programs that evaluate, improve and support Program quality need to be an integral part of the Program. They should not be organised or perceived as “add-ons”.

Two enabling conditions are particularly important for realising integration in this first sense. First, people close to the action (staff and students) must be empowered and responsible for implementing evaluation and improvement. Secondly, processes and programs need to be not only clear and systematic, but sustainable and flexible, in order to ensure longer-term effects and adaptability.

Integration is important in a second sense. Evaluation, improvement and support processes (and programs) need to be sensibly aligned and linked. For example, an evaluation process that identifies areas needing improvement in teaching or courses must be complemented by means (e.g. a program) to provide incentive and support for people to implement change.

3.7. A special note on the place of assessment of student learning

In some parts of the world, in higher education, the terms “evaluation” and “assessment” are used either interchangeably or in ways different from how they are commonly used in Australia. It is important to make clear the view adopted of the relationship between assessment and (program) evaluation.

Assessment of student learning (achievement or performance) is a fundamentally important part of the Medicine Program. Assessment and assessment results usually have two broad purposes in educational programs; formative and summative. The formative purpose of assessment is essentially to help guide student learning as learning is occurring. Summative assessment is more concerned with providing an overall judgement of the level of student achievement at various significant stages in the student’s progress in a program (typically at the end of stages). Both formative and summative assessment is relevant to program quality. Formative assessment seeks to improve the quality of student learning outcomes and experience, and summative assessment provides aggregated measures of student learning, as well as the quality of learning and what standards have been achieved.

By providing aggregated measures of student learning, summative assessment can have a particularly valuable place in program evaluation (and improvement). In fact, there is reference in the literature to this being a third major purpose of student assessment (e.g. Balla and Boyle, 1994; Bowden and Marton, 1998).

The Program Evaluation and Improvement Strategy fosters the use of appropriate summative student assessment results as part of the evidence base for improving and reporting on quality in the Medicine Program.

4. Pragmatics

The Strategy in Practice (Part 2) communicates the actual processes and other means for evaluating, improving and supporting aspects of the Program, as they are developed. The following are some of the broad but important pragmatic matters that guide the development and implementation of the Strategy.

Implementation needs to be within resource constraints.

Key responsibilities need to be vested with people running the Program - “top down” or “inspectorial” approaches are not part of the Strategy.

There needs to be a strong practical focus on improvement, support and good information for reporting – not on research for the sake of research. Program evaluation and improvement in practice is not to be a “paper generating” exercise.

Support programs are needed that enable more explicit and effective incentive and recognition for Faculty staff, including those having conjoint/clinical appointments.

Development and implementation need to be sensitive to the workloads and other demands on students and staff. Obtrusiveness and overkill are to be avoided.

There is a need for a leadership and coordination role and group to ensure good practice, and the sensible and coherent development and implementation of the PEI Strategy (see section 5).

5. The Program Evaluation and Improvement Group (PEIG)

The PEIG is an important part of the PEI Strategy. The group is a key enabler for the continuing development and implementation of the Strategy. This document was developed and will be continually improved by the PEIG.

Terms of Reference for the PEIG have been developed and these provide details of its purpose, goals and ways of operating. The following elements briefly summarise the key characteristics of the PEIG.

5.1. Overall Purpose

The overarching purpose of the PEIG is to develop a leading-edge Program Evaluation and Improvement Strategy and set of enabling processes that:

• are integral to the Medicine Program;
• are sustainable;
• encompass four main Program Quality Aspects; and
• contribute to international recognition of the Medicine Program in terms of evaluation and quality assurance.

5.2. Membership

The PEIG has a membership of a maximum of eight people appointed by the Dean of the Faculty of Medicine and is chaired by a senior academic appointed by the Dean.

Membership comprises academic and clinical staff, a person with expertise in program evaluation and quality improvement, invited student representatives, and senior professional appointments to support the group.

5.3. Reporting Line and Key Relationships

The PEIG reports directly to the Senior Associate Dean of the Faculty. It is represented on the Curriculum Development Committee and works closely with other relevant Faculty committees and teams (e.g. Phase Committees, Clinical Learning and Assessment Committee).

6. Summary

The Program Evaluation and Improvement Strategy is a key and integral part of the undergraduate Medicine Program at UNSW. Primarily, the Strategy enables systematic and continual improvement in all key aspects of the Program. The Strategy places particularly high value on the quality of student experience and learning, and the experience, capabilities and support of Faculty staff.

ln addition to its primary developmental focus, the Strategy provides a means for building a strong evidence base that can be used by the Faculty for reporting on the quality of the Program to key stakeholders, such as the AMC.

Finally, implementation of the Strategy makes a major contribution to the Faculty achieving an international reputation for excellence in the Medicine Program.

7. References

Balla, J. and Boyle, P. (1994). Assessment of student performance: A framework for improving practice. Assessment and Evaluation in Higher Education, 19(1), pp.17-28.

Bowden, J.A. and Marton, F. (1998). The University of Learning.London: Kogan Page.

Harvey, L. (2003). Student Feedback. Quality in Higher Education, 9(1), pp.1-20