Introductionand Methods Overview
Kirkpatrick Level 4 evaluations seek to measure the degree by which targeted outcomes and changes in performance are attributed to application of knowledge and skills gained as a result of the training.At the individual level, results evaluations are not particularly challenging. However, the ability to determine which results and organizational impact occurred due to training participation make measuring results at the organizational level challenging.Thus, changes should yield measurable results that directly contribute to the organization’s mission and goals (USOPM, 2011).
Initially, thePreparedness and Emergency Response Learning Centers(PERLC) Evaluation Working Group developed different instruments for conducting Level 4 evaluations, including a partner interview guide, a protocol for conducting key informant interviews, and set of indicators to measure change in emergency preparedness and response systems. The purpose of developing the different instruments was to provide instruments applicable to various data collection settings. However, the instruments later evolved to one consolidated impact evaluation tool.
The Level 4 Impact Evaluation Tool was designed to evaluate activities of the PERLC. Specifically, it assesses effectiveness and impact of the PERLC learning products. The data collection tool is organized into the following sections: information on the organization that received and/or partnered with a PERLC for a learning product; number of trained staff; descriptors of the learning products provided by the respective PERLC; PERLC partnership experience; sections for measuring systems change according to 13 Public Health Preparedness (PHEP) Capabilities; and lessons learned.For each of the PHEP capabilities, functional activities with corresponding indicators are provided.The majority of the indicators were adapted from those developed by the Centers for Disease Control and Prevention's(CDC’s) Division of State and Local Readiness to monitor and evaluate awardees of the Public Health Emergency Preparedness Cooperative Agreement during the period July 2013 - June 2014 (CDC, 2011). A 5-point Likert scale is used to measure the degree of system change, according to the 13 PHEP Capabilities, before and after the PERLC partnership.
The survey questions were written to assess the impact of PERLC activities and trainings on organizations, specifically; however, users may refine questions to meet distinctive organizational needs and various methods for data collection, accordingly. For example, the survey tool questions and appropriate prompts may be used to generate responses in a focus group setting. Regardless of the desired data collection method employed, the tool is adaptable to fit an organization’s unique needs.
