Date of Award

August 2020

Degree Type

Dissertation

Degree Name

Doctor of Philosophy

Department

Engineering

First Advisor

Naira Campbell-Kyureghyan

Committee Members

Razia Azen, Wilkistar Otieno, Michael Lovell, Hamid Seifoddini

Keywords

adult learning, guessing behavior, post-test assessment, pre-test assessment, prior knowledge, training effectiveness

Abstract

NOVEL APPROACH IN MEASURING TRAINING EFFECTIVENESS

by

Thomas Samuel

The University of Wisconsin-Milwaukee, 2020

Under the Supervision of Dr. Naira Campbell-Kyureghyan

Conducting training for employees is a popular method, in industry, to increase awareness and competencies of individuals that helps start the journey towards a change in performance. Globally, fiscal investments in the order of hundreds of billions of dollars annually are made by organizations and governments to train employees in various concepts. Of the various types of training programs, safety related training is specifically important as it increases awareness of the work risks to employees and plays a critical role in reducing safety incidents in the workplace. This has considerable societal and organizational impacts as reduction in safety incidents reduces mortality and injury rates among workers, improves their work environments and benefits the organizations as they have reduced costs and happier employees. Due to the level of investment made and associate positive impacts it is important to ensure that there is an acceptable return on investment for the training provided and that the training is effective.

As with any measurement method, training evaluations have gaps in their ability to determine if the participants are guessing or if they truly have learned the concept. Additionally, the measurement and reporting of training effectiveness can be improved to help industry trainers and organizations quickly determine which of the concepts trained were truly learned and what changes or countermeasures need to be implemented to the content or the delivery to help improve the effectiveness.

The goal of this research is to improve the assessment of training effectiveness by quantifying the effect of guessing and accounting for participant prior knowledge of the concepts. This was achieved by conducting post-hoc analysis on training assessment data collected from 1,474 participants in three major utility industries and quantifying the effect that the inclusion of the IDK option has on learning in a pre-/post-test assessment model by introducing the concept of a Control Question (CQ). The results showed that there was a statistically significant reduction of 28% in the use of the IDK option in the post-test compared to the pre-test for all questions including the CQ. Research was conducted into methods to determine the best and least learned concepts by the participants in the training. The Assessment of Training Effectiveness Adjusted for Learning (ATEAL) method was developed to adjust learning for participant prior knowledge and any negative impact they might have experienced due to the training. The ATEAL method was validated using scenario analysis and simulations and its performance was compared to the most popular metrics (Total Percentage Correct or Post-Pre Percentage Correct) currently used to report training effectiveness. The questions that were administered to the participants were grouped into safety concepts to determine which were the best and least learned concepts in the training for the different training groups and industries. The results of the ATEAL method were compared to the results reported by the commonly used metrics and detailed investigations into the merits and gaps of each method were conducted. It was observed that the ATEAL method performed better at identifying the concepts that were the best learned while compensating for prior knowledge and poor experience during the training. An additional advantage of this method is that the ATEAL method is not limited to MCQ assessments and can be used in any situation with score-based pre-/post-training assessments.

The knowledge gained from this research will enable trainers and organizations to design training assessments that make better use of the IDK option by understanding that it does reduce guessing behavior in the pre-test assessment, but it does not reduce participant guessing in post-test assessments. Including a Control Question in the pre- and post-tests assessments can be helpful with generating estimates of the probability of guessing and allow better estimates of true learning and training effectiveness. As a result of this research a method (ATEAL) was developed to allow trainers to quickly and accurately identify the concepts that the participants need further training on and where improvement to the training is required. A newly introduced concept of Training Effectiveness Matrix can be used for visual assessment of whether the trainees exhibited prior knowledge for a certain concept, if there was evidence of guessing or if the participants experienced any other possibly non-positive effects as a result of training. The results of this research enable the development and implementation of countermeasures to improve the training for the participants and thereby offer some guidance for increasing the training effectiveness. Additionally, these methods can be much more broadly adopted to any environment where there is a transfer of knowledge and thus has far reaching benefits across numerous industries and organizations, as well as for various training and assessment styles.

Download

Share

COinS