Abstracts
Résumé
Cet article rend compte d'une expérience d'évaluation d'un programme de formation offert à l'intention de propriétairesdirigeants de PME québécoises par le ministère de l'Industrie, du Commerce et de la Technologie de la province. La recherche utilise le modèle de Kirkpatrick qui propose d'évaluer un programme déformation selon quatre niveaux distincts: réactions, apprentissage, comportements et résultats. Deux cent quatre-vingt-un participants ont été rejoints par téléphone au moins un an après la tenue de leur séminaire. L'étude documente d'une façon descriptive les différents niveaux d'impact du programme de formation. Elle apporte un éclairage nouveau sur une opération rarement effectuée qui présente de nombreuses difficultés d'ordre théorique et opérationnel.
Abstract
Leaders of Quebec small businesses have access to a large number of management training programs offered by private and public agencies. Those who enroll in such programs hope to generate positive results for their business. However, the real impacts of the programs are not well documented. This paper is an account of an empirical evaluation of a training program offered to owner-managers of small businesses by the ministere de l'Industrie, du Commerce et de la Technologie (MICT) of the Province of Quebec.
CONTEXT OF THE STUDY
In 1988, the MICT offered 28 different seminars, 18 of which were aimed at owners of industrial firms, and 10 at owners of commercial and service companies. The topics discussed covered a wide range of subjects presented in workshops going from basic seminars (such as, financial and human resources management) to advanced seminars (such as, "just-in-time" management techniques and strategic planning). Seminars are in the form of one-day workshops with 10 to 15 participants.
A workshop leader, generally a consultant, university professor or businessman recognized for his competence, is responsible for the transmission of information using audio-visual materials. He introduces many practical examples and encourages communication between the participants, who are then invited to discuss their real-life business problems and to prepare concrete action plans to be implemented when they return to their organization.
THE KIRKPATRICK MODEL
A review of the literature in training program evaluation indicates that this operation is rated as essential by all authors in the field. However, it is only rarely carried out in practice. Even when an evaluation is attempted, it is often limited to measuring the participants' reaction to the seminar since it presents a number of theoretical and operational difficulties. However, our review of the literature enabled us to locate a model which is accepted as an authority by training professionals. The Kirkpatrick model proposes an analysis of the effectiveness of a training program according to four levels: reactions, learning and behaviours of the participant and results for the organization.
METHODOLOGY
From September 1986 to June 1987, the MICT seminars attracted some 3000 people. The sample is made up of about 10% of the population, namely 189 leaders of industnal businesses and 92 leaders of commercial businesses, making a total of 281 in all. Data was gathered by phone in March 1988, approximately one year after the seminar was held, using a questionnaire based on the Kirkpatrick model.
RESULTS
The participants' reactions towards the seminar are the first evaluation level in the Kirkpatrick model. On the whole, the MICT seminars seem to have been well rated by the participants. The satisfaction level is very high with respect to content and format as well as teaching and logistic support for the sessions. The second evaluation level, learning, is an attempt to determine to what extent the information transmitted in the seminars was learned and mastered. The seminars seem to have been well assimilated by the participants, since 86,9% of them state that they are completely or moderately in a position to use what they have learned during the session.
The third evaluation level aims to shed light on the changes undergone by the participant, new behaviours and projects undertaken as a result of the new knowledge. Changes in the participant himself as a result of the seminar were identified. Two hundred and thirty-eight participants generated 306 reflections that were classified under three headings: (1) Awakening, Increase in Knowledge and Awareness (33,7%), (2) Changes in Values and/or Priorities (33,3%), (3) Changes in Behaviour (33%).
The seminars had a concrete impact in terms of new projects. In fact, 51,2% of the respondents state that the knowledge acquired gave rise to a concrete project or activity in their enterprise. They mentioned 198 activities or projects which have been classified into nine categories. They are: (1) New Systems or Methods (38,9%), (2) Changes to Already Existing Systems or Methods (27,3%), (3) Analysis and Planning Activities (8,6%), (4) Establishment and Start-up of a New Unit (8,6%), (5) Training Activities (6,1%), (6) Hiring of Personnel (3,5%), (7) Purchase of Equipment or Real Estate (1,5%). Three per cent of the respondents were unable to specify the exact nature of their project.
The fourth evaluation level of the Kirkpatrick model uses different performance indicators to measure the effects of new projects and behaviour patterns on the organization. One hundred and fourty-four respondents listed 205 consequences for their own enterprise. These consequences were classified according to the type of impact.
They are positive for the enterprise in all but a few cases (4). Negative impacts are considered as temporary or compensated by positive ones in other functional spheres of the enterprise. The consequences for the enterprise are: (1) Turnover/ Sales (13,7%), (2) Profits (9,8%), (3) Costs (12,7%), (4) Management/Control (20,5%), (5) Personnel (18,5%), (6) Product (7,3%), (7) Market (6,3%), (8) Undefïned or Non-measurable Impact (9,3%), (8) Negative Impact (2,0%). CONCLUSION
Based on the results of the study as a whole, it is possible to summarize the impacts of the seminars on the participants and their businesses as being positive or even very positive. The program of the management seminars offered by the MICT seems to be a valuable tool for spreading information among small business managers whose need for training has been confirmed by several studies. The Kirkpatrick model was found to be a valuable tool in this evaluation. It is systematic, complete and intuitively logical. Moreover, it was relatively easy to operationalize. The instrument developed for this study is economic and flexible, since the survey may be done by phone. To conclude, the Kirkpatrick model is an adequate response to the theoretical, methodological and practical challenges of training program evaluation.