project evaluation

Evaluating AROMA

Background

This project provided general Risk Management (RM) Continuing Professional Development to rural obstetricians and anaesthetists. Rural specialists highlighted this topic as an area of concern and for improved learning against a background of increased indemnity and litigation from patients.

Project Aim

This project aimed to provide skills, information and networking to rural obstetricians and anaesthetists focusing on RM and related issues. Learning was done in two ways, self-directed through distance learning web-based materials and via videoconferencing. Materials developed during the life of this project were all new and original in content, emphasising the context of health care delivery in Australia and within a rural setting.

Project Content

Modules were developed for videoconferencing covering aspects of RM and skills that specialists might use in assessing their own RM and problem solving in relation to errors and adverse events. The content emphasis was on the specialist understanding his or her own risk and operating within the context of the team and health care organisation. Different levels of experience and exposure to RM varied amongst participants so that content had to be flexible and relevant to many levels of understanding and participation. The mix of private and public hospital work experience also shaped the responses and ways in which specialists interacted with content and facilitators.

Participants

The videoconferencing component of the project consisted of three sessions of two hours (four sessions in the Victorian trial). The videoconference series ran five times on a state-by-state basis to enable participants to interact with their local peers. The videoconference series ran in:

• Victoria (Trial)
• New South Wales
• Queensland
• Western Australia, South Australia, Northern Territory and Tasmania
• 'Catch-up' series for those unable to attend the series for their state

Participation in the project varied broadly in line with the number of rural specialists in each state.

Outcomes

The main outcome of the project was the creation of networking within a collegial environment where specialists could learn from each other, identifying areas of good practice in RM and establish strategies for improving local practice.

Our evaluation aimed to understand whether there was a change in skills and attitudes after participating in the project. We were successful in contributing to changes in self-reported knowledge in some aspects of RM such as root cause analysis. The impact on attitudes towards RM however was variable, this was once again self-reported,

Wider and unanticipated benefits of the project were participants learning how to use videoconferencing; understanding the perspective of others; what types of data to utilise when investigating adverse events; managing incidents

Development of resource materials and tools for recording RM activity are now available to all specialists regardless of their participation in the project and regardless of location.

Changes in skills and knowledge about risk management: we could see small changes in attitudes towards different aspects of risk management. The largest changes in attitude have been around questions that imply that risk is random and that nothing can be done about it. Through the project we have demonstrated through the use of case studies and through reflection on practice that by taking a systems approach, risk can be managed and dealt with in a proactive way.

Participation in risk management activities and root cause analysis: has been achieved through group working on problems during events and afterwards through using tools at local level

Participation in peer review: this was enjoyed by specialists, especially through the use of the case studies. Issues encountered; concern about the technical skills of individuals in the case studies; the litigious nature of patients even where there has been excellent clinical practice and multidisciplinary group work made problem solving and resolutions about each adverse event easier

The self-assessment tool has been used and completed by both O&G’s and anaesthetists, from this we have been able to gauge where participants are at with risk management at a local level. Reoccurring themes include: team working and communication, equipment and supplies issues and patient information giving and concordance.

Limitations of the Project

This project relies on participant’s self reported knowledge and skills being improved. This means that we have a subjective measure of whether participants have improved their skills and knowledge. Risk management is a wide subject area; it is challenging to teach key concepts via videoconferencing, rather than visiting sites to see what is happening on the ground. Participants also get frustrated that much of what they are learning is conceptual and needs to be translated into practice at their local level.

Conclusion

The project has raised the profile and contextualised the relevance of RM to anaesthetists and obstetricians based in rural practice. The content and skills gained from the project can be viewed as a springboard, where specialists have gained the confidence and knowledge to go back to their own organisations to ask questions about managing risk and handling patient expectations and adverse events.

The project provided good value for money as it delivered high quality content to a large number of specialists, close to home in a convenient time. The lessons learned from the project have been used to improve both content and process of delivering RM to rural specialists. New topics and spin off work has been identified from the current project that will be followed up in 2005.

© RANZCOG