Programs within the occupational health service must be evaluated regularly to ensure that they are cost-effective, successful in meeting their objectives, and managed optimally. Programs that are not successful or that meet no real need should be phased out. Programs that once met a real need might have to be adapted as needs change. Every program can be improved, but blind interference with a program in the absence of solid information is more likely to hurt than to help.

A methodical, objective evaluation provides management with the information it needs to continue, increase, or withdraw support from a program. It provides participants with a guide to their own performance and how to improve their services. It provides the client using the service with a voice to indicate whether their needs were met. A good evaluation takes planning and preparation from the onset of the program. This chapter outlines a general approach to evaluating individual programs and projects within an occupational health service.

The process of evaluation is invaluable for answering important questions about the delivery of the services. The principles of evaluation are simple to know and often exceedingly difficult to apply. This is because programs are usually set up to provide a specific service other than as a controlled experiment. The evaluation component is seldom an overriding consideration in the design of a program and indeed is often an afterthought. Consideration should be given in advance of beginning the program to the information that will be needed to evaluate that program. Records and information systems can then be designed to retrieve and process the data quickly, efficiently, and without disrupting normal activities. An evaluation that requires a halt to operations, is disruptive, and can only be achieved at great expense is unlikely to be carried out at all or will be resented by all participants.

An evaluation is carried out to answer several basic questions, among them:

1. Is (or was) there a need for this program?
2. Is the program meeting the need for which it was designed? Is it meeting other needs?
3. Are the same people for whom the program was designed actually using and benefiting from the services being provided? If not, why not?
4. Is the cost of the program reasonable for the services being provided? Where are savings possible and where might more investment pay off?
5. How effective and how efficient is this program compared to other, similar programs?
6. What changes should be made in the program which will make its operation more effective and efficient in the future?
7. Do the recipients of the program's services and the sponsors of the program understand its goals, feel satisfied with its services, and like the program? If not, why?

An informal evaluation based on the "feel" for the program on the part of someone involved in its implementation or operation may be very astute and perfectly adequate. It is more likely to be uncritical. A formal evaluation by an outsider with an objective viewpoint is best. The person or team performing the evaluation should be somewhat detached from its operation, especially if the stakes are high. Evaluation is always subject to bias but bias can be controlled by collecting as much objective data as needed, to recognize potential sources of bias before conclusions are drawn, and to balance points of view by obtaining input for all interested parties.

Programs for the provision of health services are seldom true experiments. A true experiment could involve, at the least, an experimental group receiving the treatment and a control group treated identically except that the treatment would be withheld. This is almost never possible in an occupational health program. The closer to an experimental design an evaluation method can be, however, the more objective and therefore less biased it is likely to be. Study designs which come close to a true experiment are called quasiexperimental. Quasiexperimental evaluation methods are based on comparing the experience of program participants with one of three benchmarks: 1) a similar comparison group, which is not identical but is matched closely enough to yield useful information over the same time period, 2) a before-after comparison, in which the group experience is compared before the program being evaluated began or in its earlier phases or 3) an after-only comparison group, in which the outcome of the group served by the program is compared to that of another very similar group not served by the program.

Each of the methods has serious limitations. Before-after comparisons cannot account for changes during the same period which are not related to the program, such as changes in the incidence of a disease in the community, changes in education and public attitudes as the result of news media attention, and changes in the composition of the workforce as the result of personnel turnover. Before-after comparisons and after-only comparison groups may be misleading because members of the groups may be self-selected. Individuals who cannot tolerate a job may simply quit and find another. Both the comparison group and the after-only comparison group models depend on close matching between the groups of all characteristics affecting the outcome other than the program itself. A major limitation of all three approaches is that they are poorly suited to comparisons of very small groups. Despite their limitations, these quasiexperimental models are standard techniques in the evaluation of programs.

A set of versatile techniques are adaptable to small groups and thus are potentially very powerful for the evaluation of occupational health programs and for the evaluation of the effectiveness of interventions in the workplace. The simplest of these techniques is called time series analysis. Time series analysis is an adaptation of the before-after comparison approach. Applied to programs, time series analysis may deal with specific organizational groups (departments or work teams), plants, or small companies. The measurement may include such variables as frequency of absence from work, reported injuries, or number of complaints of low back pain. To be valid, the changes in trend must be consistent for the different groups but the same intervention must have been introduced at different times. Comparison groups are not necessary. Time series analysis has many advantages. It can be applied to individuals or to groups. It is applicable to any clinical setting in which the subject or patient can be followed over time. The technique is suitable for relatively small numbers of subjects and does not require the intervention to be instituted for all subjects at the same time. It can be used when the intervention cannot be withdrawn or is irreversible. Time series analysis does have some drawbacks, however. It cannot be used when all subjects will receive the intervention at one particular time, because the variation in the timing of the intervention is what controls for outside influences on the outcome. The technique requires repeated measurements before and after the intervention. If the measurement is very subjective (such as measurement of satisfaction), depends on learning or degree of training or the age of the subject, time series analysis may not be applicable because a change may occur over time regardless of the intervention. Time series analysis is not easily analyzed by statistical tests. Despite these drawbacks, time series analysis can be extremely useful for the program manager.

Selection of an appropriate method depends on what information is accessible and whether a comparison group is available. No model is perfect for all purposes, but the most valid model should be chosen that will use the pertinent information available. Generally speaking, the most valid model is a comparison group, followed by time series and before-after comparison. After-only comparison should only be used if no other approach is possible, as in the use of programs begun in the past in which no attempt has been made to collect information systematically. The validity of a model is only part of the story, however. The data must also be accurate and the measurements meaningful.

The selection of a variable to measure in order to evaluate a program or intervention depends on the setting and the nature of the process. Four categories of measurements or comparisons may be made:

1. Outcome. Outcome measurements reflect the result of a program or intervention in terms of the actual effect on the individuals or the group. Outcome variables should be clearly associated with the intervention or program (such as blood lead in a lead control program or audiometric testing in a hearing conservation program), standardized and interpretable at the current state of knowledge (the test called "sister chromatid exchange" would not be appropriate since the relationship between the test result and actual genetic damage to humans is not fully understood), and relatively insensitive to outside factors (plant productivity reflected by overtime worked would not be useful).

 

2. Behavioral. Behavioral measurements reflect the degree to which a program has changed individual behavior patterns in a way likely to affect the outcome, such as compliance with safety procedures, eating or smoking habits, self-referral to an employee assistance program, and the change in comprehension and knowledge of a problem after an educational program.

 

3. Criteria. When a program is designed to function in a certain way, criteria can be written to describe how the process should work. The actual performance of the program can then be checked against these criteria. For example, in evaluating an occupational hearing conservation program one might ask, "Is exposure data routinely compared to hearing test results?" "Is the workplace regularly monitored every x week and when the process is changed?" "Is a selection of personal protective devices made available to workers?" "Does the soundproof booth used for audiometric testing conform to the applicable standards recommendation?" "Does the personal hearing protection device have a Noise Reduction Rating of x or more?" To be valid, such criteria must never be written with the specific program being evaluated in mind or it will be too biased. The criteria given in the audit questionnaire in Appendix 1 are examples of generally applicable criteria. Chapter???? presented an exampled of a protocol, from which such criteria are relatively easily drawn. Chapter 31 discussed standards of performance and quality of care.

 

4. Satisfaction. To ensure full compliance and cooperation, a program must satisfy the needs and expectations of the employee and the employer. A survey can determine whether the user has felt satisfactorily and courteously treated and where problems have arisen can be very useful.

These measurements are all useful in different ways. Combined, they piece together a total picture of the program as a whole and spot problems which can be solved through changes in the structure or procedures of the program.

Effective evaluation depends on good record-keeping and the cooperation of all concerned. The following are some practical measures which facilitate the effective evaluation.

1. The need for a given program should be clearly documented before a program is begun. Baseline data is essential as is the selection of the appropriate measurements to be employed.

 

2. The record-keeping system should be systematic and easy to use. All pertinent data should be recorded in an easily retrievable format, whenever possible on computer or at least in a standardized form, and kept in a central location. The amount of data collected which does not directly relate to the program should be kept to a minimum to facilitate handling.

 

3. Criteria to be used for evaluation should be formulated well before the evaluation is conducted, by someone other than the project staff. Table 32.1 presents a set of criteria developed for evaluation of the performance evaluation of an advocate presenting client's cases before worker's compensation boards on appeal. The success rate of the appeals was not a good guide to performance because too much depended on the merits of the case and the policy of the board. Rather, the evaluation was based in this instance on process criteria applied to a review of client records.

 

4. Subjective response in matters such as user satisfaction should be considered very carefully. Often expressions of dissatisfaction or lack of understanding signal grave problems which lie below the surface.

At some point in the future, perhaps all medical practices will monitor and evaluate their experience using computers. At present this is difficult but becoming easier as new techniques are developed that can be adapted for office use. Evaluation allows the physician or nurse-manager to pinpoint problems and to act to correct them. It allows the service to adapt by adding equipment, staff, or space or by cutting back in some areas for more efficient operation. It permits the physician and nurse to plan for continuing education of the staff based on the needs of the practice or weaknesses in their own preparation. It helps reduce liability by highlighting problem areas which require better control, safer procedures, referral, or further training on the part of the provider.

The methodology of evaluation rests on comparing experience against a benchmark, usually the experience of the immediate past, a comparison group, predicted results, or expectations. The experience can be expressed in terms such as patient visits, case fatality, test results positive, recovery rates, dollars expended, or other quantifiable terms. A set of criteria is established against which the experience is compared, preferably using statistical analysis which takes into account random error and variability. These criteria are broadly classified as relating to process (how the care was provided) or outcome (what the result of care turned out to be). For example, one may choose to evaluate how frequently routine blood pressure checks were performed (process) or how well blood pressure was controlled in hypertensive patients (outcome). A useful set of criteria to begin with is given in Appendix 1 in the form of an audit questionnaire.

1. Case Management (based on a review of case records)

1. Clear statement of reason for encounter and essential issues of case involving workers' compensation (WCB claim.

 

2. Assessment of client follows logically from data recorded; alternative assessments equally plausible from the data are acknowledged. Such assessments to include: health status, level of function, mental state, and special needs.

 

3. Status of client as claimant in WC claim or appeal process is clearly identified and stated on chart, updated as it changes.

 

4. Referral of client to health and social services providers is recorded, justified, and appropriate given assessment.

 

5. Follow-up and subsequent developments are clearly recorded.

2. Services in Worker Education (based on student evaluations and a review of teaching materials).

 

1. Need for a particular educational program is identified and program is designed with characteristics of learners in mind.

 

2. Opportunity for feedback from learners is provided on quality of presentation, relevance of material, and appropriateness to their situation.

 

3. Educational programs are provided in a cost-effective manner, targeting larger groups and drawing sufficient numbers to justify the commitment.

The first step in evaluation is to obtain a profile of the service. This might mean checking appointment sheets or keeping track of first reports to determine the frequency of new patients with occupational disorders and the distribution of cases by problem (musculoskeletal injuries, back strain, dermatitis and so forth by employer in the case of groups or clinics (which companies are using the service and what industries do they represent?), by occupation of employee (are all such patients truck drivers or is there a mix?), by service (acute care, consultation, periodic screening and so forth), and by geographic area (are most patients coming from one part of town?). The second step is to identify changes by comparing similar data from previous years or by continuously monitoring utilization of the service.

Once the basic descriptive data are available, other methods of analysis can be applied to develop the evaluation. For example, a sample of medical records can be audited to determine how cases were handled and how they were resolved. Financial records can be used to determine whether payment was complete or whether a few cases ran up disproportionate charges. Groups with a common occupation or exposure, such as welders or asbestos workers, may be examined to determine if they are showing consistent findings and whether screening evaluations are up-to-date. Cases of nonoccupational problems may be reviewed to see whether consistent criteria were used in certifying absence from work or fitness to return to work.

The problem of evaluating the effectiveness and cost of programs is made much simpler by gathering appropriate data at the outset for later comparison. The evaluation outline in this section is designed for occupational services providing services to a number of employers.

The essential first step in designing an evaluation is to be clear on the purposes of the exercise. Employees and sponsors of a program will understandably wish to show its progress to their advantage. This can distort the picture and obscure identification of changes that must be made. The usual objectives for evaluating occupational medicine service are the following:

1. To compare the cost to the employer of the program with alternative ways of providing the services: Cost.

 

 

 

2. To determine whether employees perceive the program as meeting their needs: Acceptance.

 

3. To determine whether employers perceive the program as meeting their needs: Satisfaction.

 

4. To determine which aspects of the program had the greatest effectiveness and at what relative cost: Specification.

The ultimate use of the evaluation is to modify the program to optimize Effectiveness, Acceptance, and Satisfaction by minimizing Cost and minimizing Specification.

Specific objectives are the yardstick for evaluation. They may be grouped under each purpose in the form of an explicit general statement of the goal and then as specific objectives:

1. Effectiveness

 

0. To protect workers against health and safety hazards.

 

0. To significantly reduce the frequency of occupational injuries and illnesses in the community or among client employers.

 

1. To significantly reduce the severity of those occupational disorders that do occur.

 

 

1. To provide adequate treatment to workers who have sustained occupational injuries and illnesses.

 

0. To significantly reduce the frequency of disability after occupational disorders.

 

1.2.2 To provide rapid and appropriate care to the impaired worker.

 

2. Cost

 

0. To deliver services at a reasonable cost.

 

0. To hold constant or reduce the total cost to the employers of medical services.

 

1. To deliver more comprehensive medical services at the same cost to the employers.

 

 

2.1.3 To deliver services at a significantly smaller cost per employee than would be the case were services obtained on a piecemeal basis.

 

2.2 To reduce the cost to the employers of secondary costs related to medical services.

 

2.2.1 To reduce employer insurance premiums.

 

2.2.2 To reduce employer assessments for workers' compensation.

 

3. Acceptance

 

3.1 To provide services in such a way that the workers subjectively feel that their needs are being met.

 

3.1.1 To establish a degree of confidence in care delivered on the part of the employees.

 

3.1.2 To establish a sense of overall satisfaction on the part of the employees with the personal attention of the clinic staff.

 

3.2 To ensure that workers' preferences and reactions are heard by the staff.

 

4. Satisfaction

 

4.1 To provide services in such a way that the employers feel that their needs are being met.

 

4.1.1 To establish confidence in care delivered to their employees.

 

4.1.2 To establish overall satisfaction on the part of the employer with the attention of the clinic to the employer's special needs and desires.

 

5. Specification

5.1.2 To determine the specific per employee cost of each program.

Any number of other objectives could be formulated but these are the essential ones which lend themselves to measurement. The design of the evaluation then must identify outcome measurements for the objectives and compare them against one of three standards:

1. A control or comparison group
2. Experience prior to participation in the program
3. Arbitrary expectations and derived criteria.

In the real-world setting of contracting for services, a nonparticipating company is hardly likely to release its balance sheets and offer its employees for interview. Thus, a control group is not realistic except in the case of similar companies which choose different components of the program for their employees within the same facility. This is not a common situation. The criterion of improvement compared to performance before participation assumes that no major unrelated activities which affected health are implemented during the study period. For this reason, the study period must be kept brief, perhaps three years after entry into the program. Arbitrary criteria are least desirable but may be the only alternative for some relatively subjective measurements such as Acceptance.

Outcome measurements must be quantifiable and accessible. Clinic-acquired data si most useful with supplementary data obtained from the employer necessary to assess preparticipation costs. Specific outcome measurements for each objective might be as follows, tabulated by participating company and calculated per 1000 employee-hours:

1. Effectiveness

1.1 Protection of employees

2. Adequacy of treatment

 

0. Claims to workers' compensation and rate of awards in the year preceding and the third year of participation in the program.

2. Cost

1. Cost of care

 

0. Cost to employer per employee per component of program (e.g., noise control, industrial accident clinic, health hazard evaluation) and total cost during last year of participation.

 

1. Coverage of individual employees as determined by number of programs and staff hours expended per unit cost per employee. (No direct comparison possible with preparticipation costs but expansion of services will be obvious if present).

 

2. Comparison of individual treatment costs with usual and customary changes in community.

 

 

2. Costs of coverage and liability

 

0. Insurance premiums before and during last year of participation.

3. Acceptance

1. Employee acceptance

 

Questionnaire responses for employees using clinic program components during first six months (covering previous arrangement and expectations of new services) and last six months (covering interim experience and subjective rating of care) of employer participation during study period.

 

2. Employee acceptance of program components

4. Satisfaction

4.1 Employer satisfaction

4.2 Employer satisfaction over program components.

5. 5. Specification

The data accumulated for the foregoing would then be examined by program component and analyzed by cost-effectiveness, cost/acceptance, and cost-satisfaction. The employees of a company are not independent in attitudes and response to services. Each program should be evaluated on the basis of its success or failure in meeting objectives for the majority of participating companies, not on the basis of all employees served.

This is a program evaluation, not a controlled epidemiologic study. Inevitable problems of evaluating programs which exist in a competitive and political milieu. By anticipating data needs at the outset and designing a general strategy for evaluation before the initiation of service, conflicts and friction with program personnel can be avoided. Program evaluations solve important problems in the real world. It is in the real world that one identifies and manages occupational disorders.