Integrating Data Instruments into Clinical Practice

One issue that MHCD has looked at recently in improving care of mental health consumers, is around the use of data instruments that Evaluation and Research Department uses to collect information on consumer recovery, and how to translate this back to the consumers, resulting in more insight about their treatment progress?  The Evaluation and Research Department uses an instrument called the Consumer Recovery Marker (CRM) and the Recovery Marker Index (RMI).  The CRM is a 15 item survey, filled out by clients and measures “active growth/orientation, hope, symptom interference, sense of safety and social networks” and is described as “consumer’s perception of their mental health recovery” (Deroche, Olmos, Hester, McKinney 2007: 1).  The RMI is a 7 item survey filled out by clinicians that includes such items as “employment, education/learning, active/growth orientation, symptom interference, engagement, housing” and substance use (added since inception), and is described as measuring “indicators usually associated with individual’s recovery, but are not necessary for recovery” (Deroche, et al. 2007: 1).  In combination, these two instruments allow for multiple perspectives on the consumer’s recovery, as well as collect information to make evaluation possible, as well as translating progress back to MHCD’s various shareholders.  The Evaluation and Research Department also utilizes a tool, the Recovery Profile (RP), which combines all the data collected from the CRM and RMI and puts them in an interpretable form, through use of line and bar graphs, showing averages of all scores, etc.  The question that MHCD faces is how to provide this information to clinicians, as well as consumers, so that it can be used to make clinical decisions around consumer treatment, as well as offering insight to the consumer about their recovery process?  In considering this issue, literature review has shown a number of factors that come into play.

Dilemma for Clinician and Organization

Another perspective is to consider the dilemma that clinicians feel, as a result of data instruments being introduced into clinical practice, which represents a switch to more standardized, or evidence based practices.  An article by Broom, Adams and Tovey (2009) looks at this issue within the healthcare and in particular, oncology practice.  This article describes using evidence based practices within medicine and states the challenge is to adopt these principles, while still maintaining “professional autonomy, clinical judgment, and therapeutic integrity” (Broom et al. 2009: 192).  I think some of clinician fears around using instruments, is that it would minimize some of their intuitive experiences with the consumers, as well as doesn’t allow for their unique talents as clinicians to shine through.  In other words, with the mental health field, a lot of a consumer’s story is shared through narrative forms, treatment plans, intakes, histories, etc. and so there might be some hesitance initially in how that can be translated into more quantitative forms, or using data instruments.   From this study, it was found that executive management was more likely to be in support of using evidence based medicine as it minimizes clinician error and is more science, objective based, and clinicians were more in opposition as they felt it takes away from some of their expertise, uniqueness as individuals (Broom et al. 2009).  The challenge is to try to find a balance, where the clinician’s unique talents can be represented and acknowledged, as well as having more information at the clinician’s disposal, in making decisions around the consumer’s treatment.

Another aspect of this dilemma is to consider the strain placed on an organization in trying to satisfy reporting needs of its stakeholders, as well as using that data in a way that improves its programs.  Carman (as cited by Hoole and Patterson 2008) reports that 65% of nonprofits engage in formal program evaluation, 95% report to their boards and 90% experience site visits from their funders (3).  The problem brought up by Hoole and Patterson (2008) in regards to this is that most of data is just collected, not used to actually improve programs.  This is most likely due to conflicting shareholder demands, as well as having support and funding for data collection, being able to translate this importance to managers, as well as staff across the organization (Hoole and Patterson 2008).  With nonprofit organizations, depending on funders, their outcome goals can often reflect interests of the shareholders, and thus are not intertwined with mission of the organization (Hoole and Patterson 2008).  Basically, the answer to this for Hoole and Patterson (2008) are for the nonprofit to work more with the shareholders on integrating outcome requirements with that of their own internal mission or goal statements.  This holds true for MHCD, as well in that they currently are making efforts to integrate data already being collected to satisfy Medicaid funding, or other state, private needs, in a way that can be translated to clinicians and used in clinical practice and making decisions around client care.

Structure and Funding

An article by Carman and Fredericks (2009) describes that how successful evaluation is implemented depends on “autonomy, internal structures and external relations, leadership styles and maturity” (Carman and Fredericks 2009: 3).  Carman and Fredericks also identify the Executive Director as the key into how research and evaluation is implemented into non-profits (Carman and Fredericks 2009).  This study uses three different clusters of non-profits and finds that those that are primarily funded by government, Medicaid, public funds have fewer problems of support and funding for research and evaluation (Carman and Fredericks 2009).  The following is a breakdown of funding for MHCD and was taken from the “MHCD 2009: Report to the Community” and covers the fiscal year concluded June 30, 2009.

Source                                                            Amount                                  Percentage

Medicaid                                             $23,925,630                            44.4

State of Colorado                               $12,945,194                            24.0

Client, Third Party, Pharmacy            $8,826,030                              16.4

Contracts and Grants                          5,632,041                                10.5

Interest, Rent,             Other                           $1,604,777                              3

Medicare                                             $498,286                                 0.9

Public Support                                                $444,236                                 0.8

Some other findings were the larger the organization, more likely to identify staff resistance as a problem in evaluation, data collection (Carman and Fredericks 2009).  Younger organizations were found to have more technical assistance issues with evaluation and organization; with connections to housing or community development, there are fewer problems with implementation and design (Carman and Fredericks 2009).  With MHCD being a larger organization and having a lot of support for evaluation from executive management, as well as having primary funding from Medicaid and the State of Colorado, there is already a lot of familiarity with data collection and evaluation, at least at the administrative level.  However, with it being a large organization and with multiple layers and treatment teams, residential and employment facilities, translating the information that is collected by Evaluation and Research Department to the clinicians and consumers, offers chance for more resistance.

Philosophy

            The mental health field has some differences in philosophy in comparison to medical field as well, that have made it more difficult for evidence based practices to be implemented successfully.  Rishel (2007) points out that the mental health field focuses more on clinical outcome trials or best models of treatment, rather than on prevention itself.  Rishel (2007) goes on to state two of possible reasons for this are prevention coming from a public health perspective and looking at population as a whole, which varies from a clinical approach aimed at best methods to treat those already diagnosed with mental illness.  Also, prevention methods are usually thought of as requiring larger samples and for participants to be followed for a long period of time, compared with clinical trials which are shorter and require smaller sample size (Rishel 2007).  The mental health field is also seen as being hard to evaluate in terms of outcomes, as there are really no standardized outcomes (Rishel 2007).  This becomes even more difficult with less definition attributed to non-profit organizations.  This proves true with MHCD in that most of the focus to this point has been best treatment models to integrate into clinical practice.  However, with integration of data instruments, this allows for more longitudinal data and more of a focus to prevention side. 

Clinician Feeling Towards Instruments     

A study done looking at mental health field and how clinicians feel towards data instruments was conducted by Garland, Kruse, Aarons (2003).   This study was done on mental health system in California and found that 92% of clinicians reported never using scores from data instruments in their clinical practice (Garland et al. 2003: 400).   Further, 90% identified a collecting the data as a “significant time burden” in terms of fitting into their daily work tasks (Garland et al. 2003: 400).  This data shows clinicians not putting much value into data collection, as well as thinking of it more as a burden, versus something that could be useful for them in their practice.  The article also went on to show that 55% of clinicians in reference to measures used by the instrument’s, felt they were “not appropriate, nor valid, for their particular patient population” (Garland et al. 2003: 400).  It would be a hard sell to get clinicians to buy into using some of these instruments, if they don’t believe they are valid or useful for their population.  When asked what changes the clinicians wanted to see with how the data was reported, answers were “briefer administration” or “simpler language”, as well as wishing these were presented in “narrative, as opposed to quantitative form” (Garland et al. 2003: 400).  As we can see, there seems to be a lot of doubt from clinicians around trusting the data is appropriate for their client populations, as well as being able to interpret the data, and doubting whether the data reflects anything that can be used in clinical practice.

           

            As we can see from the literature, there are multiple interests to consider when implementing data instruments into clinical practice.  MHCD is unique from a lot of non-profits in having its own internal Evaluation and Research Department, and this creates a lot of opportunity to progress forward, in terms of how clinical information is relayed to clinicians, as well as consumers receiving mental health services.  MHCD has already made a lot of steps towards helping ease this transition.  The MHCD “Recovery Committee”, which is already a committee that was in place, but is now working on how to help with this transition.  There were focus groups held with clinicians around what concerns they had with the instruments, and this information was used to make some changes to the instruments, as well as in developing trainings.  The trainings were designed to show how the data collection instruments can be interpreted, as well as how to use that information in discussions with consumers to give them incentive in participating in data collection.  MHCD also has a team made up of MHCD consumers who have taken the responsibility of going to each site and sharing with other consumers their experiences with looking at the RP, and how beneficial it is to view this data and learn more about their treatment progress.  MHCD will continue to evaluate how this integration has gone, but it has created a unique opportunity as an organization, to bring various clinical teams and consumers together, to work on getting the most out of this data that is already collected.  In the medical health field, I think we have seen a lot of growth in terms of how we can see lab work, communication with our doctors through e-mail, other electronic means, etc.  I think through this example, MHCD has shown how the mental health field can benefit from data instruments as well, resulting in more client informed, as well as clinician informed care. 

Submitted By Jim Linderman- Evaluation Specialist with Evaluation and Research Department at MHCD, as well as Sociology M.A. student at University of Colorado Denver Sociology Program. 

References

Broom, A., J. Adams, P. Tovey (2009). “Evidence-based healthcare in practice: A study of clinician resistance, professional de-skilling, and inter-specialty differentiation in oncology” Social Science & Medicine 68(192-200).

Carman, J.G., K.A. Fredericks (2009). “Evaluation Capacity and Nonprofit Organizations: Is Glass Half-Empty or Half-Full?” American Journal of Evaluation 31:84.

DeRoche, K., Hester, M., Olmos, P.A., McKinney, C.J. (October, 2007). Evaluation of Mental Health Recovery: Using Data to Inform System Change. Poster presented at the 'Culture of Data' Conference. Denver, CO.

Garland, Ann F., M. Kruse, G. A. Aarons (2003). “Clinicians and Outcome Measurement: What’s the Use?”.  Clinicians and Outcome Measurement 30(4):393-405.

Hoole, E. and T.E. Patterson (2008). “Voices from the Field: Evaluation as part of a Learning Culture”. Nonprofits and Evaluation. New Directions for Evaluation 119:93-113.

MHCD 2009: Report to the Community (can be found at www.mhcd.org)

Rishel, C. (2007). “Evidence Based Prevention Practice in Mental Health: What is it and how do we get there?” American Journal of Orthopsychiatry 77(1): 153-164.