The Simulation Module for Assessment of Resident’s Targeted Event Response (SMARTER-Shift): a novel teaching method in Emergency Medicine education

Introduction:

One of the most fundamental approaches in medical education is rooted in Halsted’s statement “See One, Do One, Teach One”. Although this emphasizes the clinical nature of medical education, it also raises concern about the quality of patient care in the educational setting (1). In recent years, this concern has grown. Simulation-based training (SBT) aims to reach a compromise between these two seemingly opposite goals (2-5). High-fidelity mannequin simulation provides medical students the opportunity for observation, evaluation, and feedback that may not occur in the clinical setting.

The Simulation Module for Assessment of Resident’s Targeted Event Response "SMARTER" methodology which was first described in 2008 is an Event-based approach to training which has been customized for using in healthcare.

Rosen showed the use of simulation in graduate medical education affords unique opportunities for increasing the quality of a resident's educational experiences (6).

Simulations are not only capable of reproducing the dynamic nature of real-world clinical experiences, but are also good news for those who blame education for poor and improper patient care (7). Some have even proposed a pedagogical framework which integrates simulation training into routine training of procedural skills for medical students (8). One approach in incorporating SBT into medicinal education is the SMARETR (simulation module for assessment of resident’s targeted event response) methodology (9), which uses an eight-step algorithm starting from Accreditation council of graduate medicinal education (ACGME) core competencies and ending in creation of a scenario script (10-12).

The SMARETR methodology provides a tool to guide learning experience and captures nearly all dynamic interactions in clinical interventions. In this process an ACGME core competency is selected (step one), based on which educational objectives are clarified (step two). Then a clinical context is chosen (step three) in which the Knowledge-Skill-Attitude (KSA) profile of good performance is defined (step four). Based on these, critical events (step five) and expected responses (step six) are designated and a measurement tool for the response is devised (step seven). In the last step all of the above are incorporated into a clinical scenario (step eight) (6). As one can see in step seven, the method can also be used as the base for education evaluation which is free of the limitations imposed by assessment at the bedside (6,13) .

Most of the available simulation technologies and most of the teaching processes that use simulation are designed to enhance procedural skills or at most, the management of a single patient, and thus lack the concept of multi-component problem solving (14-16) . In emergency medicine (EM), on the other hand, the very feature of multiple task management is deemed valuable. Using prioritizing techniques and reordering what needs to be done, an EM specialist must be able to face repetitious intrusions or different interruptions which might happen simultaneously in order to manage different tasks at the same time (17,18). When first introduced to EM, even the advocates of SMARTER declared that performance in EM is complex and covering its entirety within one scenario is difficult, impractical, and even questioned its necessity (6). Consequently, when SMARTER methodology is used, each scenario mainly focuses on a specific competency. Instead, we believe while each specific competency is important, the ability to perform in a competent manner in different dimensions simultaneously and under pressure is what separates a capable EM specialist from others. The fact that EM education has turned a blind eye to this issue means residents face additional stress and frustrations during a real shift clinical experience which drastically reduces their performance quality (19). Moreover, residents need to try and observe expert clinicians, handling multiple cases, and learn both the accuracy and timeliness of ordered medications through direct observation within the clinical setting (20). The problem with this approach is that each clinician may have his/her own attitude towards a particular treatment and this may in turn lead to a variety of treatments and a lack of homogeneity and standardized approaches.

However, it has been proposed that SBT has the potential to compensate for this drawback and help build a well-defined set of Knowledge Skill and Attitude (KSA) in the high-acuity multi-patient environment (21). In this study we describe and test the “SMARTER-Shift” as a teaching method in which several clinical scenarios based on the SMARTER approach are integrated to create a simulated setting in which several competencies are demonstrated alongside the ability to multitask.

In this study we take that methodology one step further and use it to design multiple patient scripts hoping to enhance multiple knowledge and skills including multitasking.

Methods:

This was a mixed study composed of two distinct parts. First the development of multi-patient SMARTER scenarios rooted in ACGME core competencies to cover major realms of knowledge, skill and attitude within the selected field, alongside being able to demonstrate the ability to manage multiple patients simultaneously. Second a quantitative before and after study aiming to assess the ability of this method to achieve its teaching objectives.

Scenario development

The eight steps of SMARTER (previously discussed) were used to create five patient scenarios for each SMARTER-Shift (6).A ninth step was added in which each patient’s introduction into the ED and also all the critical events of different patients were integrated into a single timeline. This was done so that resident would benefit more than one proper step to take and prioritize between these tasks. Also after finishing one task, the resident may be expected to resume a task which he/she had left unfinished; therefore the ability to task-switch was also demonstrated. A third possible situation was an unimportant interruption which needed to be deprioritized by the resident. Expected responses (step six) and measurement tools (step seven) pertinent to these new skills were also included in the assessment sheet. As the leader resident in each simulated shift was responsible for coordination of other team members, another demonstrated and rated skill was the ability to manage a treatment team.

Use of simulation technologies

Alongside the use of simulated case scenarios, simulation technologies where incorporated into cases to better teaching and assessment of resident skills. For this purpose five mannequins were available in each SMARTER-Shift. Based on the context of the scenarios these included a standard intubation mannequin, a rhythm and CPR simulation mannequin, a suturing mannequin, a pediatric intubation mannequin, a central line placement mannequin, a chest tube placement mannequin, basic airway management kit, a peripheral intravenous placement kit, a ventilator, digital radiologic results, EKG strips, and/or an ultrasonography devise. The equipment was initially covered by sheets and each mannequin or apparatus was revealed on the specified time on the simulated shift timeline and residents were expected to assess, perform procedures, or interpret data accordingly. 

Data gathering and assessment of KSAs

The residents’ knowledge and skills were assessed using pretest and posttests. Our pretest composed of a written and an OSCE exam both of which had been used for the evaluation of residents in our department on previous occasions in the related topic. Then the shift residents were asked to complete an anonymous questionnaire regarding their opinions on the use of SMARTER-Shifts as a teaching method and its ability to prepare them for a clinical shift.

In order to prevent the pretest acting as an educational vector, the posttest was delayed to 4 weeks after holding the simulated shift. The posttest again composed of both written and OSCE examinations on the relevant topic with questions covering the same domains but not the exact items covered on the pretest. In this study we aimed to evaluate the effects of the SMARTER-Shift on resident education, not as an assessment tool. But still assessment tools were devised as dichotomous checklists to help students realize their strengths and weaknesses during debriefing.

Statistical Analysis

The statistical software SPSS V21 was used for data analysis of the quantitative values. Student paired T-tests and standard deviations were used to analyze the pre/posttest scores.

Results:

The SMARTER-Shift

A sample SMARTER-Shift is described in Table-1. The critical events have been designated specific times on the multiple scenario timeline. Each shift was conducted with the participation of five residents all in their first year of postgraduate studies. One resident was determined as the lead physician. The instructor acted as the confederate nurse while the other four students could also be assigned tasks by the lead physician as nurses or staff. The data pertaining to each patient was announced by the instructor on specified times.

Table 1.  Sample SMARTER-Shift in the trauma ward

After a briefing of about 10 minutes the original scrip ran without interruption regardless of resident responses for 45 minutes. During this time the instructor filled out the evaluation sheet regarding KSA competency. The four assisting students were also asked to note every error which is different in opinion with the lead physician, or with questions. During the shift, the lead resident was asked to document orders and progress notes as would be expected in an actual shift. Table-2 describes the presentation of each of the five simulated patients into the sample SMARTER-Shift described in Table-1.

Table 2. Final scenarios of SMARTER-Shift

After the shift and a 10 minute rest, an hour long debriefing session was held in which the strengths and weaknesses of the lead resident in responding to critical events, questions and opinions were discussed with the group. After that the same scenario script ran with a different lead physician, this time allowing for interruptions for questions by the instructor and aiding residents, real-time feedback, and clarifications upon request. The aiding residents were also given the chance to practice the included procedures. Overall each SMARTER-Shift lasted between 4 to 5 hours.

Quantitative analysis

Presented findings are based on five SMARTER-Shifts that were held for residents of emergency medicine at Mashhad University of Medical Sciences during the summer 2016. In total, 25 residents participated in the study voluntarily. Most participants were male (n=19, 76%). The mean age of participants was 32.93 (ranged 25-45). The students’ cumulative scores for knowledge and skills improved significantly before and after the SMARTER-Shift (P<0.001). Overall satisfaction with the SMARTER-Shift as indicated on a ten-point visual analogue scale was 9.96±0.20. Table-3 summarizes results of the pretest and posttests.

Table 3. Mean and standard deviation of residents’ scores before and after intervention

Most residents (96%) agreed that the SMARTER-Shift can boost their ability to think of multiple clinical issues simultaneously. Nearly all participants (96%) agreed that this method can speed up their management of critically ill patients during a clinical shift; while 84% of residents agreed that it can reduce their stress when faced with multiple patients during a real clinical shift. Moreover 84% agreed that the SMARTER-Shift can help them prioritize responsibilities and medical orders during a clinical shift. Also 88% of residents agreed the SMARTER-Shift was successful in transferring required knowledge for management of patients within the emergency department. As for clinical skills, 80% of residents believed the SMARTER-Shift was a more effective way to gain knowledge and learn clinical skills than a real clinical shift; but when they were asked which setting they find more effective for practicing the gained knowledge and skills only 52% of them believed the SMARTER-Shift was more effective and 48% still found a real clinical shift to be more effective when it comes to practicing previously acquired KSAs. 

Discussion

In this study we describe a new educational method which aims to provide a comprehensive experience for residents of emergency medicine which closely resembles their experience in the clinical setting.

The step by step development of a multiple patient scrip is described and a sample script is presented. Cumulative scores of skill and knowledge for resident improved significantly (P<0.001) before and after the course. Almost all students (96%) preferred SMARTER-shift to a clinical shift for teaching purposes while only 52% preferred it for practicing the skills they had already obtained. 

Evans and colleagues conducted a study “A Comprehensive, Simulation-Based Approach to Teaching Clinical Skills: The Medical Students’ Perspective “in the year 2014. The results of the study showed that a simulation curriculum can enhance the ability to manage acute clinical problems and switch well to the clinical experience (3).

We call this method “The SMARTER-Shift”. In this method we take advantage of previously described SMARTER methodology, and available simulation technologies to enhance resident experience in multiple knowledge and skill domains, while affecting their attitude hopefully and positively. Our results showed that the SMARTER-Shift, although time consuming, is effective in enhancing knowledge and skill proficiency and is rated highly successful in achieving all of its goals by residents. Also a score of 9.66 out of 10 is an impressive affirmation of participants’ satisfaction.

The changing face of technology and new expectations from residency programs around the world demand for development of new and more efficient methods of education. On one hand, the past two decades have seen the rise of simulation as a means of teaching and assessing physicians (22) through eliminating ethical concerns in order to rise clinical medical education (1). On the other hand, the core competencies defined by ACGME, which have been integrated into many residency training programs (12, 23), and are considered as corner stone of for broadening the medical education program.

The SMARTER methodology which was first described in 2008 is an Event-based approach to training which has been customized for use in healthcare (6,13).

Here we have updated the process by adding a ninth step. In this final step several patient scenarios are integrated into one scrip and create opportunities for residents to learn and display competency in multitasking and multiple patient care.

The ability to manage multiple patients simultaneously is a well-recognized characteristic of the Emergency Physician (EP) (18).There are several conscious and subconscious techniques that EPs use to achieve this responsibility. While real multi-tasking is rare, most EPs use task-switching, prioritizing, task-stacking, and willingness to welcome intrusions and interruptions, to do this (19).  It has been suggested that failure to learn these skills is a contributing cause to EP stress, burnout, fatigue, and sleep deprivation. Currently most EM residency programs do not officially include the teaching of these skills, while EM certifying institutions test them (17,19). In our study, 84% of participants agreed that the SMARTER-shift was successful in reducing their stress in real life clinical shifts and the same number found it successful in transferring the required knowledge and skills to residents.

It appears that the most common method for residents to gain the skills of managing multiple patients simultaneously is through direct observation of their supervisors which entails many limitations. Although primitive, this is the method that most EPs have acquired these skills and its importance must not be overlooked. If performed in a timely, graceful manner by the faculty, this onsite learning experience can be invaluable (20). Another method for teaching multi-tasking skills is multidisciplinary team simulation with scenarios focusing on multitasking, and multiple patient care (20). This can be achieved alongside the use of SBT or as a roundtable with the participation of faculty members and other students (24). Obviously the latter requires one to employ imagination for a fuller experience.

With the expansion of simulation technology, multiple encounter simulation scenarios have also been proposed for this purpose. These have been previously reported to create life-like educational settings for EM residents (21,24)but to our knowledge none have focused attention on teaching the skills of multitasking in the emergency department. Most literature have focused on alleviation of possible medical errors and improving team work. One review of SBT for trauma teams found that most of these studies have focused on the lower levels of education such as reaction, and learning, while fewer studies have looked at behavior and result levels. Yet, still improvement in knowledge, skill, and team work was generally significant in all levels after SBT (25).  Here we have combined the benefits of SMARTER methodology with SBT and multiple case scenarios. Although we too assess the lower levels of education in a before and after design too, our findings are very promising and high satisfaction of residents is impressive.

Another point that must be emphasized is the importance of debriefing as part of SBT (20,21). It provides a situation in which the residents can learn from their mistakes and reflect on their errors without the concerns of medico-legal and ethical issues. In our study we designed preliminary dichotomous checklists which helped the instructor determine which responses were presented in a timely manner and which were not. This was the basis for a one hour debriefing session which followed the initial 45 minute shift.

Although in our study we used the SMARTER methodology which has been tailored to meet the educational needs and objectives of residents, we believe that our methods may be adjusted to serve as a teaching platform in other clinical fields as well. The satisfaction of nursing students with simulation based training has been described extensively (26-29). We believe that a similar methodology based on nursing competencies may be an important aid in teaching nursing students, especially those training for advanced education and more specific positions in the emergency department.

In this study we described and tested the SMARTER-shift, as a novel educational method for teaching the care of multiple patients. Our results showed that while this is a time-consuming task the process is feasible and effective in improving knowledge and skills, with almost unanimous satisfaction amongst the residents taking part.

Ethical considerations 

Ethical issues (Including plagiarism, informed consent, misconduct, data fabrication and/or falsification, double publication and/or submission, redundancy, etc.) have been completely observed by the authors.

 Acknowledgement

The authors would like to thank the interns who participated in their study.

Financial support

This article is the result of a research project by Mashhad University of Medical Sciences with the code of 940313, sponsored by Mashhad University of Medical Sciences.     

Conflict of interest

The authors declare that they have no competing interests.