Speaker 1: The training function is interested in assessing the effectiveness of training programs. Training effectiveness refers to the benefits that a company and trainees receive from training. Benefits for trainees may include learning new skills or behaviors. Benefits for the organization can include increased sales and more satisfied customers. A training evaluation measures specific outcomes or criterias to determine if the benefits of the program exist. Training outcomes or criteria refer to measures that the trainer and the company use to evaluate training programs. To determine the effectiveness of training, evaluation needs to occur. Training evaluation refers to the process of collecting the outcomes needed to determine whether training is effective. The information from the needs assessment should be used to develop an evaluation plan. Companies invest in training because learning creates knowledge. Often, it's this knowledge that distinguishes successful companies and employees from those who are not. Research summarizing the results of studies that have examined the linkage between training and human resource outcomes show that training provides extraordinary benefit to competitive advantage of organizations. The influence of training is largest for organizational performance outcomes and human resource outcomes and weakest for financial outcomes. Strategic training is more strongly related to organizational outcomes when it's matched with the organization's business strategy and capital intensity. Training evaluation provides a way to understand the investments that training produces and provides information needed to improve training. Formative evaluation refers to the evaluation of training that takes place during program design and development. That is, formative evaluation helps to ensure that 1. The training program is well organized and runs smoothly and 2. Trainees learn and are satisfied with the program. Formative evaluation provides information about how to make the program better. It usually involves collecting qualitative data about the program. Qualitative data includes opinions, beliefs, and feelings about the program. Formative evaluations ask customers, employees, managers, and subject matter experts their opinions on the description of the training content and objectives of the program design. The formative evaluation is conducted either individually or in groups before the program is made available to the rest of the company. Trainers may also be involved to ensure the time requirements of the program are met. As a result of formative evaluation, training content may be changed to be more accurate, easy to understand, or more appealing. The training method may be adjusted to improve learning, such as providing trainees with more opportunities to practice or give feedback. Also, introducing the training program as early as possible to managers and customers helps in getting them to buy into the program, which is critical for their role in helping employees learn and transfer skills. Formative evaluation involves pilot testing. Pilot testing refers to the process of previewing the training program with potential trainees and managers or with other customers, persons who are paying for the development of the program. Pilot testing can be used as a dress rehearsal to show the program to managers, trainees, and customers. It should also be used for formative evaluation. Summative evaluation refers to an evaluation conducted to determine the extent to which trainees have changed as a result of participating in a training program. That is, have trainees acquired knowledge, skills, attitudes, behavior, or other outcomes identified in training objectives? Summative evaluation may also include measuring the monetary benefits, also known as return on investment or ROI, that a company receives from the program. From the discussion of summative and formative evaluation, it's probably apparent to you why a training program should be evaluated, but let's take a look at the key factors. To identify the program's strengths and weaknesses. This includes determining if the program is meeting the learning objectives, if the quality of the learning environment is satisfactory, and if transfer of training to the job is occurring. To assess whether the content, organization, and administration of the program, including the schedule, accommodations, trainers, and materials, contribute to the learning and the use of training content on the job. To identify which trainees benefit most or least from the program. To assist in marketing programs through the collection of information from participants about whether they would recommend the program to others, why they attend the program, and their level of satisfaction with the program. To determine the financial benefits and costs of the program. To compare the costs and benefits of training versus non-training investments, such as work redesign or a better employee selection system. To compare the costs and benefits of different training programs to choose the best program. Information gained from the training design process is valuable for training evaluation. Training evaluation must be considered by managers and trainers before training has actually occurred. Information gained from the training design process shown in this figure is valuable for training evaluation. The evaluation process should begin with determining training needs. Needs assessment helps identify what knowledge, skills, behaviors, and other learned capabilities are needed. Needs assessment also helps to identify where training is expected to have an impact. The next step in the process is to identify specific, measurable training objectives to guide the program. The more specific and measurable these objectives are, the easier it is to identify relevant outcomes for evaluation. If the needs assessment was done well, the stakeholders' interests likely overlap considerably with the learning and program objectives. Once the outcomes have been identified, the next step is to determine an evaluation strategy. Planning and executing the evaluation involves previewing the program, formative evaluation, as well as collecting training outcomes based on the evaluation design. The results of the evaluation should also be used to encourage all stakeholders in the training process to design or choose training that helps the company meet its strategy. The six categories of training outcomes, reaction outcomes, learning or cognitive outcomes, behavior and skill-based outcomes, effective outcomes, results, and return on investment. Reaction outcomes refer to trainees' perceptions of the program including the facilities, trainers, and content. Reaction outcomes are typically collected via a questionnaire completed by trainees. Cognitive outcomes are used to determine the degree to which trainees are familiar with the principles, facts, techniques, procedures, and processes emphasized in the training program. Self-assessments refer to the learners' estimates of how much they know or have learned from training. Skill-based outcomes are used to assess the level of technical or motor skills and behaviors. Skill-based outcomes include acquisition of learning of skills, skilled learning, and the use of skills on the job, known as skill transfer. Skill transfer is usually determined by observation. Trainees may be asked to provide ratings of their own behavior or skills known as self-ratings. Effective outcomes include attitudes and motivation. Effective outcomes can be measured using surveys. Results are used to determine the training program's payoff for the organization. Return on investment or ROI refers to comparing the trainees' monetary benefits with the cost of the training. Appropriate training outcomes need to be relevant, reliable, discriminative, and practical. Criteria relevance refers to the extent to which training outcomes are related to the learned capabilities emphasized in the training program. The learned capabilities required to succeed in the training program should be the same as those required to be successful on the job. Criterion contamination refers to the extent that training outcomes measure inappropriate capabilities or are affected by extraneous conditions. Criteria may also be deficient. Criterion deficiency refers to the failure to measure training outcomes that were emphasized in training objectives. Reliability refers to the degree to which outcomes can be measured consistently over time. Discrimination refers to the degree to which trainees' performance on the outcome actually refers to true differences in performance. Practicality refers to the ease to which the outcome measures can be collected. One reason companies give for not including learning, performance, and behavior outcomes in their evaluation of training programs is that collecting them is too burdensome. There are a number of reasons why companies don't evaluate training. Learning professionals report with that access to results and the tools needed to obtain them are the most significant barriers. Access to results is often determined by the extent to which managers and leaders understand the need for evaluation and support it. From our discussion of evaluation outcomes and evaluation practices, you may have mistaken impressions that it's necessary to collect all five levels of outcomes to evaluate a training program. While collecting all five levels of outcomes are ideal, the training program objectives determine which ones should be linked to the broader business strategy. It's important to recognize the limitations of choosing to measure only reaction and cognitive outcomes. Remember that for training to be successful, learning and transfer of training must occur. Which training outcome measure is best? The answer depends on the training objectives. How long after training should outcomes be collected? There's no accepted standard for when the different training outcomes should be collected. In most cases, reactions are usually measured immediately after training. Cognitive transfer is evident when learning occurs but skills, effective outcomes, or results are less than at pre-training levels. This discussion of evaluation designs begins by identifying the alternative explanations that the evaluator should attempt to control for. Threats to validity refer to factors that will lead an evaluator to question either the believability of the study results or the extent to which the evaluation results are generalizable to other groups of trainees and situations. The believability of study results refer to internal validity. These characteristics can cause the evaluator to reach the wrong conclusions about training effectiveness. An evaluation study needs internal validity to provide confidence that the results of the evaluation, particularly if they're positive, are due to the training program and not to another factor. Because trainers often want to use evaluation study results as a basis for changing training programs or demonstrating that training does work, it's important to minimize the threats to validity. One way to improve the internal validity of a study result is to first establish a baseline or pre-training measure of the outcome. Internal validity can be improved by using a control or a comparison group. The Hawthorne effect refers to employees in evaluation studies performing at a high level simply because of the attention they're receiving. A random assignment refers to assigning employees to the training or comparison group on the basis of chance alone. A number of different designs can be used to evaluate training programs. The pretest-posttest refers to an evaluation design in which both pre-training and post-training outcome measures are collected. The pretest-posttest-with-comparison group refers to an evaluation design that includes trainees and a comparison group. Time series refers to an evaluation design in which training outcomes are collected at periodic intervals both before and after training. The strength of this design can be improved by using reversal, which refers to a time period in which participants no longer receive the training intervention. A comparison group can also be used with time series designs. The time series design is frequently used to evaluate training programs that focus on improving readily observable outcomes such as accident rates, productivity, and absenteeism that vary over time. The Solomon four-group design combines the pretest-posttest-comparison group and the posttest-only-control group design. There is no one appropriate evaluation design. An evaluation design should be chosen based on an evaluation of factors. Cost-benefit analysis is the process of determining the economic benefits of a training program using accounting methods that look at training costs and benefits. Training cost information is important for several reasons. To understand total expenditures for training including direct and indirect costs. To compare the costs of alternative training programs. To evaluate the proportion of money spent on training development, administration, and evaluation as well as to compare money spent on training for different groups of employees like exempt or non-exempt employees for example, and to control costs. There is an increased interest in measuring the ROI of training and development programs because of the need to show the results of these programs to justify funding and to increase the status of the training and development function. Most trainers and managers believe that there is a value provided by training and development such as productivity or service improvements, cost reductions, time savings, and decrease employee turnover. However it's important to keep in mind that ROI is not a substitute for other program outcomes that provide data regarding the success of a program based on trainees' reactions and whether learning and transfer of training have occurred. However ROI is also useful for forecasting the potential value of a new training program choosing the most cost-effective training method by estimating and comparing the costs and benefits of each approach. The process of determining ROI begins with an understanding of the objectives of the training program. Plans are developed for collecting data related to measuring these objectives. The next step is to isolate, if possible, the effects of training from other factors that might influence the data. Developing evaluation outcomes and designing an evaluation that helps isolate the effects of training. Because ROI analysis can be costly, it should be limited only to certain training programs. Metrics are used to determine the value that learning activities or the training function provide the organization. One way to understand the value that learning activities or the training function provides is through comparisons to other companies. Big data refers to complex data sets developed by compiling data across different organizational systems including marketing and sales, human resources, finance, accounting, customer service and operations. Volume refers to the large amount of available data. Variety includes the large number of sources and types of data that are available. Velocity refers to the huge amount of data that's being generated and the speed at which it must be evaluated, captured, and made useful. Big data can come from many different sources including transactions, business applications, emails, social media, smartphones, and even sensors embedded in employee's identification badges or company products. The goal of big data is to make decisions about human capital based on data rather than intuition or conventional wisdom which likely lead to incorrect conclusions and recommendations. Big data can be used for many purposes including to evaluate the effectiveness of learning and development programs, determine their impact on business results, and develop predictive models that can be used for forecasting training needs. Using big data requires the use of workforce analytics. Workforce analytics refers to the practice of using quantitative methods and scientific methods to analyze data from human resource databases, corporate financial statements, employee surveys, and other data sources to make evidence-based decisions and to show that human resource practices including training, development, and learning influence important company metrics.