Community Mental Health Journal

38 (2): 119-128, April 2002

Copyright © 2002 Human Sciences Press, Inc.

All rights reserved

Beyond Components: Using Fidelity Scales to Measure and Assure Choice in Program Implementation and Quality Assurance

Robert I. Paulson, Ph.D.

Regional Research Institute, Portland State University, P.O. Box 751, Portland OR 97207; paulsor@rri.pdx.edu

Rachel L. Post, MSW

Portland State University

Heidi A. Herinckx, MSW

Portland State University

Patrick Risser, BA

Portland State University

Abstract

Fidelity scales have become an accepted part of intervention research. Initially, fidelity scales focused on critical components of an intervention. In this paper we argue that the next generation of fidelity scales should include key process variables such as choice. Since choice is an essential element in all empowerment and recovery driven intervention models, a fidelity scale for an enhanced version of the Individual Placement and Support (IPS) supported employment model that incorporates choice as a fundamental component was developed as part of a SAMHSA community action grant. The process for developing the choice component and the dimensions measured are also described.

Keywords

fidelity scales, supported employment, consumer choice, empowerment, recovery

Article ID: 369951

Beyond Components: Using Fidelity Scales to Measure and Assure Choice in Program Implementation and Quality Assurance

Fidelity scales have become an accepted part of intervention research for models serving those diagnosed with severe mental illness. Initially, fidelity scales focused on critical components of an intervention. In this paper, we argue that the next generation of fidelity scales should include key process variables such as choice. Since choice is an essential element in all empowerment and recovery-driven intervention models, we have developed a fidelity scale for an enhanced version of the Individual Placement and Support (IPS) supported employment model (Drake,1998; Drake, Becker, Clark, & Mueser, 1999) that incorporates choice as a fundamental component (for choice section of fidelity scale, see Table 1 at end of paper).

Fidelity scales were developed in response to two major problems with effectiveness trials of psychosocial rehabilitation programs, particularly case management. The first was the lack of specification of the program being researched, so meaningful comparisons between studies could not be made. Secondly, if a program failed to produce the anticipated results, it was not always clear whether the failure was due to poor implementation or to flaws in the original program theory and design. Furthermore, once a program’s effectiveness had been documented, the program was often rapidly sought after for dissemination or replication. However, local adaptations are frequently necessary, which raises the question: “At what point do local adaptations change the model sufficiently that it no longer can be called the same intervention?”

As community-based intervention research evolved and replications produced contradictory results, it became clear that the next step was to establish which components of an intervention were the most critical in producing successful outcomes. Concerns were raised by the initial designers of programs when key elements were left out of replications and less successful outcomes were reported. Many programs, for example, called themselves Assertive Community Treatment (ACT) programs (Witheridge, 1991; Bond, McGrew, & Fekete, 1995), yet did not adhere to essential elements of the ACT model such as team assignment of cases, “in vivo” services, assertive outreach, staff/consumer ratios of 1:10. In the development of the first ACT fidelity scale, McGrew and colleagues (1994) asked 20 experts in ACT to identify and then rate the essential elements of ACT. This scale was later refined (Teague, Drake, & Ackerson, 1995) and used to show that programs that had higher fidelity were, in fact, more effective in producing the desired outcomes (McHugo, Drake, Teague, Xie, 1999). Similar scales were later developed for supported employment and IPS to measure program implementation and drift (Bond, Becker, Drake, Vogler, 1997; Bond, Picone, Mauer, Fishbein, & Stout, in press).

Recently, fidelity scales have become an essential tool in multi-site studies like the ACCESS program and the CMHS Supported Housing Initiative. In a multi-site study, there are major differences between the organization and financing of mental health and rehabilitation systems across the sites. Also, local cultures often produce idiosyncratic elements that influence service delivery models. It is, therefore, particularly important to be able to say that the phenomenon being studied is truly the same across all the sites, or at the very least to specify the differences.

Thus, fidelity scales have become an important tool in intervention research to ensure that the intervention being tested is fully implemented. Fidelity scales are also used by service providers to prevent program drift over time. However, all published fidelity scales to date have focused on program characteristics and essential components, such as organizational structure, staffing patterns, and service delivery characteristics.

For years there has been considerable discussion about client-centered services (Rapp & Poertner, 1992), and inherent in the notion of client centeredness is the belief that clients should have maximum choice about their care. Choice is essential in providing individualized services and ensuring a match between consumer characteristics, wants, and needs, and program offerings. Promoting consumer choice and preference is essential to consumer empowerment and rehabilitation (Carling, 1995). Plans based on the perceptions of professionals or family members are less likely to be successful than those based on the choices of the recipient of those services. Carling (1995) states, “The core to empowerment is returning to consumers the responsibility for choices about their lives and their lifestyles” (p. 287). This end can be met, Carling contends, by ensuring that the following elements are incorporated into services: a) providing consumers with real options; b) assisting and empowering consumers to define their own preferences; c) allowing consumers to make tradeoffs and decisions based on their preferences; d) providing access to supports, regardless of what choices individuals make; and e) providing successful role models. Recipients of services are more likely to build solid trusting, collaborative working relationships with those who listen to their wants, and help them to access those services that will assist them in meeting their own goals.

Furthermore, guidelines developed by the National Association of State Mental Health Program Directors (NASMHPD, 1989) emphasize the importance of recognizing that mental health consumers have the expertise and knowledge to contribute to mental health services. They recommend that states include consumers in program development, policy formulation, quality assurance, systems designs, education of professionals, and service provision. Additionally, in a report to Congress entitled From Privileges to Rights: People Labeled with Psychiatric Disabilities Speak for Themselves, the National Council on Disability (NCD, 2000) recognizes coercion as a barrier to seeking out mental health treatment by consumers and as incompatible with preservation of self-determination rights. Indeed, the NCD recommends that public policy move in the direction of a totally voluntary community-based mental health system that safeguards human dignity and respects individual autonomy.

Finally, Monahan and colleagues (1999) report findings from the MacAruthur Coercion study that indicate that the kind of pressures applied to mental health consumers entering psychiatric hospitals strongly affects perceptions of coercion among those consumers. Negative pressures that use threats and force result in a higher experience of coercion, whereas positive pressures such as persuasion and inducements do not. The amount of coercion experienced is also based on the consumers perception that others treated them with respect, acted out of genuine concern, and took time to hear their side of the story.

Although it is well accepted in all aspects of normal life that choice enhances a person’s motivation to participate in an activity, there is little controlled research in this area. It is possible that choice in and of itself could be an important variable in predicting outcomes, equal or greater to the content of the intervention itself.

Consumer choice is gaining greater attention as an essential element in the delivery of mental health services. For example, the CMHS Supported Housing Initiative developed a fidelity scale incorporating consumer choice as an inherent characteristic of the supported housing model. Choice was added as a new dimension to the more traditional fidelity scale. The value of consumer choice is also central to the philosophy of the Individual Placement and Support Model (IPS). As part of a CMHS Community Action Grant (Nikkel & Paulson., 1998) to build consensus for the adoption of an enhanced model of IPS in Portland, Oregon, the steering committee developed a fidelity scale that included a consumer choice section modeled after the type of choice questions used in the Supported Housing Initiative.

The model known as IPS+ incorporates enhancements into the Individual Placement and Support (IPS) model of supported employment services for those diagnosed with severe mental illness. This new model integrates case management services using features of the nationally recognized Assertive Community Treatment model and the IPS model, while also incorporating treatment for co-occurring disorders (substance abuse/mental illness) into services provided by a treatment team made up of both consumer and non-consumer providers.

The IPS+ model uses an employment-as-treatment philosophy to promote rehabilitation and recovery. This is achieved by assisting those served in rapid job placement in a vocation of choice, as defined by recipients of services. The provision of ongoing supports and assessments are offered and driven by the stated desires of the recipients of services, including job coaching, case management, housing and benefit supports, substance abuse treatment, and assistance with medication management.

As consensus to implement IPS+ grew, a fidelity scale for IPS+ was developed using components of the published scales for ACT, IPS and supported housing. To reflect the importance of consumer choice in the IPS+ model, a consumer choice section was also constructed. Many programs give lip service to choice and client-centered services; however, there are no measures available to determine if, in fact, this philosophy was actually being implemented. If fidelity scales are important in ensuring that essential components of an intervention are present, then they can also be used to ensure that essential processes such as client choice are being implemented as well.

Two consumer consultants were hired to work with the IPS+ project coordinator and project evaluators to develop a series of questions to measure the degree of choice incorporated in all phases of the IPS+ treatment process. This group essentially recreated what theoretically would happen to a consumer from initial contact to successful completion of the program and discussed where and how choices should be given. This resulted in a comprehensive section of 41 questions measuring consumer choice.

Six dimensions of consumer choice were identified: a) awareness/access; b) freedom to make decisions; c) service individualization; d) absence of coercion; e) belief in consumer competence; and f) respect for client confidentiality. Following is the rationale for each of these dimensions and some sample questions used in the IPS+ fidelity scale.

Awareness/Access:

Choice is a key element of IPS+, in contrast to the philosophical approaches of some other programs that rely more on professional judgment. As noted above, an essential component of the IPS philosophy is that work is therapeutic and that persons diagnosed with severe mental illness can be placed directly in jobs, without any pre-employment or job readiness services (Bond, 1998; Drake, McHugo, et al., 1996) with equal or better success. Traditionally, considerable judgment has been exercised by mental health practitioners in determining when a consumer was “ready” for employment and vocational services. Often, vocational services were not provided until the consumer was determined to be “stable enough” for such services. Consequently, access to and awareness of vocational programs is often restricted, and consumers are denied these services whether or not they may have wanted such services.

Similarly, many vocational programs insist that consumers be “clean and sober” before they can receive vocational services, despite recent evidence that substance use is unrelated to success in competitive employment (Sengupta, Drake & McHugo, 1998). In fact, it is possible that work may reduce active substance use. A meaningful job is a powerful motivator for consumers to modify their substance use, and offers financial rewards as well as a new life style and pro-social peer group.

For consumers to have true choice, they should be offered vocational services at intake. IPS+ emphasizes providing services and rapid job placement to any client as an interest in work is expressed. Therefore, the IPS+ fidelity scale asks consumers to respond to items such as:

· “How soon did someone ask you if you were interested in work?”

· “I received employment services as soon as I expressed an interest in working.”

· “I was told that I could not participate in employment services.”

Freedom to Make Decisions:

The freedom to make decisions about all aspects of one’s services and to change one’s mind about them is an obvious component of choice. Frequently, however, consumers are either given only a limited range of options regarding the kind of work in which they can participate (the famous four F’s: food, filth, foliage, and filing) and various aspects of the work environment (e.g., location, time of shift). These limitations can have an important influence on satisfaction and success on the job. Furthermore, programs may be reluctant to support a person when they change their mind about a work site because of the investment they have already made in developing the job or fear that they will hurt their relationship with the employer. Items on the IPS+ fidelity scale that are relevant to this dimension are:

· “I have the right to change my mind about what I do.”

· “My choices are respected by my team.”

· “I chose the geographic area or job location of where I wanted to work.”

· “I chose the kind of work I wanted.”

Service Individualization:

It would be hard to argue that services are client centered or that consumers have real choices if they do not receive individualized services, but rather are forced to chose among a limited number of options determined by a treatment program. Instead, the individuals’ expressed needs and desires should drive the kinds of services received. Items on the fidelity scale for this dimension include:

• “My individual needs and choices determine the services I get.”
• “I am asked what is important to me in choosing the kind of work I want to do.”
• “I am provided with the kind of work assistance that I need, when I need it.”

Absence of Coercion:

The ultimate test of individualized services is whether or not individuals receive all of the services they feel they need in order to be successful and satisfied. To assess this on an ongoing basis, the fidelity scale includes a list of service needs usually considered as part of the array of community support services that should be available. The consumer is asked the extent to which each service is needed and whether or not that service was received.

By definition, there can be no choice if there is coercion. Yet many programs practice a not-so-subtle form of coercion by making program participation or access to goods (housing) or services conditional on doing certain things (e.g., taking medication, going to therapy, receiving case management services), or abstaining from certain behaviors (e.g., abstaining from substance use). Examples of items used to assess choice without coercion are:

• “I have been coerced to do things I didn’t want to do in this program.”
• “If I refuse substance abuse treatment, I am told that I am not eligible to receive employment services.”
• “ I feel I must work in order to receive mental health treatment.”

Belief in Consumer Competence:

As noted before, professionals frequently make decisions about whether or not to offer services to an individual based on their belief in the consumer’s ability to “benefit” from such services. Hence, the staff’s belief in a consumer’s competence can directly limit choice by the selective withholding of options or services. IPS+ strives to empower consumers and to maximize their potential to achieve their individual life and employment goals. To tap this dimension, examples of items covered were:

• “Staff expect and encourage me to do things for myself.”
• “People I work with expect me to fulfill my job responsibilities just like anyone else.”
• “Every work experience I have is viewed as a success by the team.”

Respect for Client Confidentiality:

A final critical dimension of choice is one’s ability to control information about oneself. Given the stigma attached to severe mental illness, control over the disclosure of information about one’s mental illness is particularly critical as such disclosures can have very real and severe consequences for a person’s life. Disclosure can occur in direct and indirect ways and led to the development of scale items such as:

• “It is up to me whether or not I tell my employer about my diagnosis or disability.”
• “I decide whether or not program staff visit me on the job and in the workplace.”
• “Program staff respect my right to privacy on the job and in the workplace.”

The IPS+ Project decided to make the IPS+ fidelity scale an integral part of the quality assurance process in the program implementation phase. Staff and consumers will be asked to fill out the fidelity scale questions both at program start up and every six months thereafter to ensure that consumer choice is respected throughout the life of the implemented IPS+ program.

The next step for the consumer choice section of the IPS+ fidelity scale is to pilot it and test its psychometric properties. A factor analysis will be conducted to see whether the items in each dimension are empirically related to each other, and whether there is sufficient internal consistency to produce replicable and reliable results.

Most mental health interventions are value neutral. That is, they can be implemented either in an empowering, consumer-centered manner, or in a coercive manner. It is important that arguments regarding the effectiveness of interventions not become confused due to a lack of clarity regarding philosophy and values. One approach to overcoming this lack of clarity is to measure program philosophy and values as part of fidelity scales so we can better understand the importance of these elements, as they are reflected in practice and on client outcomes. The urgency of such endeavors is underscored by the current controversy over the use of coercion in ACT programs, which has led many consumers to condemn the intervention itself irrespective of how it is implemented. We chose to write this paper at this early stage not to promote this particular fidelity scale, but to suggest to the field that the next generation of fidelity scales should consider adding important process dimensions that may include operationalizing values to promote or prevent certain practices.

Table 1: Consumer Choice Questionnaire (to be filled out by IPS+ Consumers)

CONSUMER RATING

1. How soon did someone ask you if you were interested in work?	Never	At 6 months	At 3 months	At 1 month	At intake or my first contact with IPS+
2. How often were you given the opportunity to discuss your interest in work?	Never	Rarely	Sometimes	Often	Always
3. A work specialist was available to meet me to discuss my interest in work.	Never	At 3 months	At 1 month	At 1 week	Sooner than 1 week
4. I received employment services as soon as I expressed interest in working. (job search, job coaching)	Never	At 3 months after initial meeting with work specialist	At 1 month after initial meeting with work specialist	At 1 week after initial meeting with work specialist	Sooner than 1 week after initial meeting with work specialist
5. I was told that I could not participate in employment services.	Strongly disagree	Disagree	Neither agree nor disagree	Agree	Strongly agree
6. I am pursuing my goals and interests, just like anybody else.	Strongly disagree	Disagree	Neither agree nor disagree	Agree	Strongly agree
7. Staff expect and encourage me to do things for myself.	Never	Rarely	Sometimes	Often	Always
8. My individual needs and choices determine the services I get.	Never	Rarely	Sometimes	Often	Always
9. I receive services as often as I want them.	Never	Rarely	Sometimes	Often	Always
10. I have the right to change my mind about what I want to do.	Never	Rarely	Sometimes	Often	Always
11. My choices are respected by my team.	Never	Rarely	Sometimes	Often	Always
12. I have been coerced to do things I didn't want to do in this program.	Never	Rarely	Sometimes	Often	Always
13. I was invited to write my own treatment plan.	Not at all	Not very often	Some of the time	Most of the time	All of the time
14. I feel that my treatment plan reflects my goals and interests.	Not at all	Not very often	Some of the time	Most of the time	All of the time
15. I decide whether or not I take prescribed medication.	Never	Rarely	Sometimes	Often	Always
16. I'm thinking about work as part of a career path.	Almost never true	Rarely true	Occasionally true	Often true	Almost always true
17. An employment specialist meets with me at times and places convenient to me.	Never	Rarely	About half the time	Usually	Always
18. I am asked about what is important to me in choosing the kind of work I want to do.	Never	Rarely	About half the time	Usually	Always
19. The employment specialist and I discussed different types of work I was interested in.	Never	Rarely	About half the time	Usually	Always
20. In order to receive employment services, I have to see a prescribing psychiatric professional (psychiatrist or nurse practitioner) or other mental health professional regularly (representative payee or case manager).	Strongly disagree	Disagree	Neither agree nor disagree	Agree	Strongly agree
21. If requested by me, staff is available to help discuss issues with my employer.	Never	Rarely	Sometimes	Often	Always
22. If I refuse to take prescribed medications, I am told that I am not eligible to receive employment services.	Strongly disagree	Disagree	Neither agree nor disagree	Agree	Strongly agree
23. If I refuse substance abuse treatment, I am told that I am not eligible to receive employment services.	Strongly disagree	Disagree	Neither agree nor disagree	Agree	Strongly agree
24. I feel that I must work in order to receive mental health treatment.	Strongly disagree	Disagree	Neither agree nor disagree	Agree	Strongly agree
25. I was asked about my skills, strengths and interests to help me determine what kind of work I wished to do.	Strongly disagree	Mildly disagree	Agree and disagree equally	Mildly agree	Strongly agree
26. I am provided supports (including encouragement) that I need to successfully find and keep work.	Never	Rarely	Sometimes	Often	Always
27. Staff assistance is available to me when I have a crisis, including when I am at work.	Never	Rarely	Sometimes	Often	Always
28. I chose the geographic area or job location of where I wanted to work.	Strongly disagree	Mildly disagree	Agree and disagree equally	Mildly agree	Strongly agree
29. I am provided with the type of work assistance that I need, when I need it.	Never	Rarely	About half the time	Usually	Always
30. I chose the kind of work I wanted.	Never	Rarely	Sometimes	Often	Always
31. Because of where I work, people know I have a disability.	Strongly disagree	Disagree	Neither agree nor disagree	Agree	Strongly agree
32. It is up to me whether or not I tell my employer about my diagnosis or disability.	Never	Rarely	Sometimes	Often	Always
33. It is up to me whether or not I tell my co- workers about my diagnosis or disability.	Never	Rarely	Sometimes	Often	Always
34 I have the same rights as any worker at my work place.	Never	Rarely	Sometimes	Often	Always
35. People I work with expect me to fulfill my job responsibilities just like anyone else.	Never	Rarely	Sometimes	Often	Always
36. If I have needs or require accommodations on the job, I feel comfortable talking directly to my employer.	I have never felt this way	I have rarely felt this way	I have sometimes felt this way	I have felt this way often	I always feel this way
37. My work schedule (including the # of hours I work) is set by my employer and me, not by my staff.	Strongly disagree	Disagree	Neither agree nor disagree	Agree	Strongly agree
38. I decide whether or not program staff visit me on the job and in the workplace.	Strongly disagree	Disagree	Neither agree nor disagree	Agree	Strongly agree
39. Program staff respect my right to privacy on the job and in the workplace.	Strongly disagree	Disagree	Neither agree nor disagree	Agree	Strongly agree
40. Every work experience I have is viewed as a success by the team.	Strongly disagree	Disagree	Neither agree nor disagree	Agree	Strongly agree
41. I am supported if I change jobs.	Never	Rarely	Sometimes	Often	Always
42. I find my job rewarding and stimulating	Never	Rarely	Sometimes	Often	Always

References

Bond, G.R., Becker, D.R., Drake, R.E., Vogler, K.M. (1997). A fidelity scale for

the Individual Placement and Support model of supported employment. Rehabilitation Counseling Bulletin, 40(4), 265-284.

Bond, G.R., McGrew, J.H. & Fekete, D.M. (1995). Assertive outreach for frequent

users of psychiatric hospitals: a meta-analysis. Journal of Mental Health Administration, 22, 4-16.

Bond, G. R., Picone, J., Mauer, B., Fishbein, S., & Stout, R. (in press). The

Quality of Supported Employment Implementation Scale. In G. Revell, K. J. Inge, D. Mank, & P. Wehman (Eds.), The impact of supported employment for people with significant disabilities: Preliminary findings from the National Supported Employment Consortium. Richmond, VA: Virginia Commonwealth University.

Bond, G.R. (1998). Principles of the Individual Placement and Support Model:

Empirical Support. Psychiatric Rehabilitation Journal, 22 (1), 11-23.

Carling, P.J. (1995). Return to community: Building support systems for people

with psychiatric disabilities. New York: The Guilford Press.

Drake, R.E. (1998). A Brief History of the Individual Placement and Support

Model. Psychiatric Rehabilitation Journal, 22 (1), 3-7.

Drake, R.E., Becker, D.R., Clark, R.E., & Mueser, K.T. (1999). Research on the

Individual Placement and Support model of Supported Employment. Psychiatric Quarterly, 70, 289-301.

Drake, R.E., McHugo, G.J., Becker, D.R., Anthony, W.A., & Clark, R.E. (1996).

The New Hampshire Study of Supported Employment for people with severe mental illness. Journal of Consulting and Clinical Psychology, 64, 391-399.

McGrew, J.H., Bond, G.R., Dietzen, L. Salyers, M. (1994). Measuring the fidelity

of implementation of a mental health program model. Journal of Counseling and Clinical Psychology 62(4), 670-678.

McHugo, G.J., Drake, R.E., Teague, G.B., Xie, H. (1999). Fidelity to assertive

community treatment and client outcomes in New Hampshire dual disorders study. Psychiatric Services, 50(6), 818-824.

Monahan, J., Lidz, C.W., Hoge, S.K., Mulvey, E.P., Eisenberg, M.M., Rothe, L.H.,

Gardner, W.P. & Bennett, N. (1999). Coercion in the provision of mental health services: The MacArthur Studies. Research in Community and Mental Health, 10, 13-30.

National Association of State Mental Health Program Directors (NASMHP). (1989). Position paper on consumer contributions to mental health service

delivery systems. Alexandria, VA: Author.

National Council on Disability (2000). Executive Summary, From Privileges to

Rights: People Labeled with Psychiatric Disabilities Speak for Themselves. Washington, DC: Retrieved January 27, 2000 from the World Wide Web: http://www.ncd.gov/publications/privileges.html

Nikkel, R. & Paulson, R. (1998). Phase I: Planning and Implementing an IPS+

Program. Grant Application for CMHS Community Action Grant SM52481-01.

Rapp, C.A. & Poertner, J. (1992). Social administration: a client-centered

approach. New York: Longman.

Sengupta, A., Drake, R.E. & McHugo, G.J. (1998). The relationship between

substance use disorder and vocational functioning among people with severe mental illness. Psychiatric Rehabilitation Journal, 22(1), 41-45.

Teague, G.B., Drake, R.E, & Ackerson, T.H. (1995) Evaluating Use of continuous

treatment teams for persons with mental illness and substance abuse. Psychiatric Services, 46 , 689-695.

Teague, G.B., Bond, G.R., & Drake, R.E. (1998). Program fidelity and Assertive

Community Treatment: Development and use of a measure. American Journal of Orthopsychiatry, 68, 216-232.

Witheridge, T.F. (1991). The “active ingredients” of assertive outreach. New

Directions for Mental Health Services, 52, 47-64.