Lessons learned from the Victorian PHEEM project

A guide to the use of PHEEM in Australia

The Australian version of the PHEEM instrument has been used successfully to provide valuable data about the educational environment in Victorian hospitals. In the course of this project, key elements that enhance the successful use of the instrument have become evident and are outlined below.

Key element 1

Have a thorough understanding of the PHEEM instrument. To do this you will need to:

Read the original article describing the PHEEM (Roff et al);
Having an understanding of the psychometric testing of PHEEM is also advised (Boor et al) (Riquelme et al);
Understand the scoring schema; and
Review the Australian PHEEM form and become familiar with the 40 items.

Key element 2

Be clear about the Why, Where, Who, and How of using the PHEEM:

Why you are using the PHEEM instrument – eg. gathering information about various jobs / units / departments; quality improvement etc. Using the PHEEM instruments for external benchmarking is potentially counter-productive in a collaborative project.
Where to use it – jobs / units / departments;
Who will be involved – all prevocational levels or limited to one prevocational group;
Who to report the data to; and
How to collect the data. Various data collection methods yield vastly different results. The greatest return rates have been achieved by collecting data at group education sessions. A return rate of 85% is our aim.

Key element 3

Ensure organisational support for using the PHEEM. Having a local champion is also very beneficial so designate someone at the hospital to ensure that the following activities occur:

Administer the distribution and collection of PHEEM questionnaires. If data about specific jobs is being collected it will be necessary to develop a coding system to identify each job;
Distribute analysed results as appropriate;
Provide feedback to the participating units / departments and participating doctors; and
Facilitate review of the results and items for attention.

Key element 4

Clarify the following important details before commencing the PHEEM:

Extent of involvement – is it a collaborative activity involving several hospitals or just one hospital? There are benefits in working with others including the capacity to expand positive aspects of the clinical learning environment, and also in facilitating ways to address identified areas for improvement. In our experience, many areas of the learning environment that require attention are common among hospitals. See PHEEM Report (Stage 2) 2009;
How anonymity will be maintained - this requires greater consideration in smaller facilities with small numbers of prevocational doctors in each job / unit; and
Who is responsible for facilitating the central collection and analysis of the PHEEM forms and the subsequent distribution of collated data to participating hospitals.

Victorian PHEEM Program

The Victorian PHEEM project is an ongoing initiative facilitated by the Postgraduate Medical Council of Victoria and jointly conducted by Medical Education Officers (MEOs) at participating hospitals. The PHEEM instrument used in Victoria is identical to the original UK version except for minor amendments to language, which make it congruent with Australian terminology; for example, the item ‘I am bleeped inappropriately’ has been changed to ‘I am paged inappropriately’. The demographics section of the Victorian PHEEM instrument was expanded. Australian PHEEM form

Conducting the PHEEM

MEOs at participating hospitals administer the PHEEM at the end of each term rotation to junior doctors in their first, second and third postgraduate years (PGY 1, 2, 3). Completed questionnaires are collected and PMCV coordinates the processing and statistical analysis of the information, which is then reported back to the participating hospitals. Each hospital is responsible for disseminating their data as necessary to address their needs.

Hospitals have varying needs and purposes in relation to their involvement in the PHEEM project including the capacity to benchmark; and to find out what hospitals individually and collectively do well and where improvement is needed.

PHEEM scoring

Statistical analysis of the PHEEM data includes the production of mean results for an overall score ( /160) and for all 40 items (rated 0-4). An overall PHEEM score of more than 81/160 indicates a positive response, while results lower than this indicate that plenty of problems exist in the educational environment. A mean score of 3 or 4 for an item reflects positively for that aspect of the educational environment; conversely, a score of less than 2 reflects negatively.

Rollout of the PHEEM project in Victoria

The project commenced in 2006 with six hospitals involved; the Royal Children’s Hospital; Bendigo Hospital; Latrobe Regional Health; Peter MacCallum Cancer Centre; Western Hospital; and Hamilton Base Hospital.

Other hospitals that have joined the PHEEM project include: Albury Wodonga Health; Ballarat Health Services; Northern Health; Southwest Health Care; Western Health; Echuca Health; Bendigo Health; Melbourne Health and St Vincent's Health.

PHEEM Reports and Presentations from 2006-2009

Victorian MEOs Evaluate Hospital Educational Environments (Prevoc Conference, Sydney, 2007)
Evaluate Learning Environments with PHEEM: Victorian Hospital focus on 'Working with Others' (Ozzawa, Melbourne, 2008)

In 2009, it was decided that in-depth information about some of the results was required to progress the project, so five focus groups involving 70+ junior doctors were conducted. See PHEEM Report (Stage 2) 2009.

Enhancing the Educational Environment for Junior Doctors (Prevoc Conference, Gold Coast, 2009)

PHEEM Reports and Presentations from 2010-2014

PHEEM Downunder - Medical Teacher 2010
Using the PHEEM Project to improve feedback: A collaborative initiative in Victoria (Prevoc Conference, Adelaide 2013)
Using the PHEEM project to promote and support collaborative learning for Medical Educators (ANZHAPE, Melbourne 2013)
PHEEM Feedback Project 2013-14

PHEEM Reports and Presentations 2015

Feedback Tool-kit for Supervisors

A Feedback Tool-Kit for supervisors has been developed and is available on the PMCV website. The Tool-Kit has been developed in response to information from supervisors that highlighted their perceived lack of preparedness to provide feedback, especially negative feedback to address performance concerns.

View Feedback Tool-kit >>

A nuts and bolts summary of the tool-kit is also available.

View summary>>

What is PHEEM?

PHEEM is the Postgraduate Hospital Educational Environment Measure, a scored forty item questionnaire that was developed in the UK by Roff et al to evaluate various aspects of the clinical learning environment for junior doctors. PHEEM scores three domains of the clinical learning environment: perceptions of autonomy; perceptions of teaching; and perceptions of social support. Psychometric testing of the PHEEM instrument in 2007 by Boor et al concluded that it is one-dimensional and does not measure three domains; however, further psychometric testing in 2009 by Riquelme et al suggests the PHEEM instrument is multi-dimensional. PHEEM has been used successfully in several countries including England, Scotland, Chile, Japan, Denmark, Brazil, and Sri Lanka.