An evaluation of Action After Review (AAR)

Project Plan

Background

After Action Review (AAR) was developed by the USA military, and is a method of evaluation conducted during or after any project.  It allows employees and project leaders to discover and learn what happened during the project and why. It can be thought of as a professional discussion of an event that enables employees to understand why things happened during the progression of any process or project, and to learn from that experience. The AAR is a continual method of evaluation that connects past experience with future actions, and can be performed after each identifiable event within a project or major activity, and therefore becomes a live and continuous learning process.  This project is one of the first evaluations of AAR in the United Kingdom. 
The AAR is a professional discussion that includes all the participants involved in a project and focuses directly on the tasks and goals. It is not a critique and has several advantages over a critique:

  • It does not judge success or failure.
  • It attempts to discover why things happened.
  • It focuses directly on the tasks and goals that were to be accomplished.
  • It encourages employees to discover important lessons in the discussion.
  • More employees participate so that more of the project or activity can be recalled and lessons can be learned and shared.

After Action Review focuses on four key questions:

  1. What was supposed to happen?
  2. What actually happened?
  3. Why were there differences?
  4. What can you learn from this experience?

A lack of ongoing evaluation of projects expecting to change practice can be costly to any organisation due to redundant project work, and repetition of mistakes. It is suggested that a properly conducted AAR can be an influential method to use in any organization, understanding that every action is an opportunity for learning. It can become part of the communication process that educates, informs and motivates a project group, by ensuring they continue to do the right thing. It can prevent future confusion ensuring lessons are learnt from successes and mistakes.

Aim

To pilot the use of AAR within the Practice Development Team at Sheffield Teaching Hospitals NHS Foundation Trust (STHFT), to assess its value as a method of evaluation in a busy acute hospital. 

The Practice Development Team undertakes projects which affect all areas of the Trust, and potentially impact on the outcomes for patients, and/or the service, and/or the staff.  An aim of CLAHRC SY TK2A theme is to identify key success factors about getting research into practice which can help embed project learning into the organisation.    

Objectives

  1. Provide examples of evaluating AAR in practice, in different departments using projects linked to clinical practice.
  2. Identify how the process of AAR can contribute to team working and project success/failure.
  3. Identify factors related to the successful use and application of AAR related to healthcare.
  4. Provide a vehicle for CLAHRC partners to share the pilot evaluation of AAR as an ongoing assessment method for projects.

Approach

A mixed methods service evaluation approach was used to evaluate AAR from March to May 2010.   The methods were a survey, observation, and audio taping of the AAR meetings.

  • Four projects from the learning and development department, and one from CLAHRC SY were observed, ranging from 20 minutes in length to 2 hours.  e.g. A ward review of the introduction of the Inter Professional Patient Record, Team Review of Support and Leadership Programme for ward staff.   
  • The overall duration of these projects varied in length from short reviews to ongoing evaluation projects lasting over one year.
  • The AAR’s were carried out at the end of some evaluations, but mid project in others.

All participants were briefed about the purpose of AAR before the session began. All the AAR’s were observed by the same individual, notes taken, and were audio-taped with the participants’ permission.   Detailed written notes of the discussions were recorded by the observer. The audio tape was listened to afterwards for clarification and to add any points that may have been missed to the notes.   Written feedback from the observation notes and audio tape transcription was sent to the lead of the group for each of the projects who took part.

A 12 item questionnaire on the perceived value of elements of the AAR process was distributed to staff taking part at the end of each AAR.  It used a Likert scale of strongly agree to strongly disagree, and took 10 minutes to complete. There were 18 responses from 24 staff; respondents also had the opportunity to add free text comments to the questionnaire.     

The role of the facilitator in the AAR’s 

All of the AAR’s had a facilitator trained in AAR to guide the team through the AAR process.  The facilitators followed the steps and guidelines for conducting AAR’s by   setting the climate for the AAR. Ground rules for openness, honesty, and confidentiality were discussed, and the facilitator allowed and encouraged all participants to contribute.  The facilitator also aimed to keep the group on track, answering the four key AAR questions, and to keep to time.  The facilitator was independent and unfamiliar with the projects, and was therefore able to ask probing questions to the groups which helped them to reflect and recall events to add to the discussion.