So around implementation context — we all have our models so here’s my model. Actually, this model was developed by our faculty team in the Implementation Research Institute. I know that David and Larry and going to plug TIDIRH, but the faculty team when we were preparing to launch our first NIH supported training program said, “We’ve got to figure out what’s important. So together we developed this framework.

With treatment efficacy and effectiveness research we’re very familiar with what’s on the far left hand here. That is an evidence-based treatment, or that is a program. And it produces patient outcomes. So that’s business as usual. We test an intervention for clinical outcomes.

But in implementation research, we have some new ingredients. We have, first of all, the how. Those are processes or strategies, the implementation strategies. It’s in the same box here with the what because both of those are professional activities, they’re programs, they’re interventions, they require that we do something.

We also have a couple of other boxes of outcomes. The middle box are service system outcomes, that the IOM has said all health professionals better start paying attention to. These are things like the efficiency, the safety.

I think we’ve all had a family member who’s gone in for surgery, and they write with a marker on the part of the body that’s going to be operated on that says, “Operate here,” so that we don’t get the wrong knee replaced. So that concept of safety we grasp it, you know, how important that is. I work with anesthesiologists who talk about how important it is to make sure that the patient not wake up during surgery, and I thought, “Oh my –” I didn’t even think of that as a possibility. But so there are a lot of safety issues that we’re all over when it comes to physical medicine. But there are safety issues in our fields too.

There are issues of equity, disparities, patient centeredness, and timeliness. And I worry a lot about timeliness in social work and mental health in that people wait a long time before they come for help. I know that’s true in your field too. Stigma, thinking why I ought to be able to manage this, this keeps people from coming quickly. And we are not very timely in the delivery of evidence-based or newly developed programs.

So in addition to business as usual — testing an intervention for a patient outcome, the IOM says let’s look at systems of care. And we had a great session yesterday on health services research.

There also are unique outcomes in implementation, and I’m going to talk about those in a few minutes, and why it’s important that we have distinct implementation outcomes. So the heart then of implementation research is looking at this area, the implementation strategies, the service system outcomes, and the implementation outcomes.

Now, hybrid designs have meant it’s not either/or. But I think my talk and a couple other talks later it may help you to know that we are talking about a different kind of research.

So then the context — all of that was surrounded by context. So that’s the first construct that I want to help unpack this morning.

Some of the things we need to figure out about service or treatment context are: who pays, who are the policymakers? There are administrators, there are researchers, clients, patients and families, front line providers, support staff. And we need to think where are each of these stakeholders when it comes to implementation; are they on board, are they happy with the status quo, who’s chomping at the bit for change, and who’s resisting?

I think the concept of stakeholders is an important way to begin to unpack what we mean by context.

There are sometimes demands to implement. Illustrating a very discreet strategy, an implementation strategy: — I remember two or three years ago I was working with a hospital, the director of a certain kind of services that I won’t name, and he was very excited to tell me that they were going to start implementing a new guideline on Monday morning in this unit. And I said, “Oh, that’s great. Tell me about that, what have you been doing, what’s your process?” And he was puzzled by that. And he said, “I’m issuing an email at 8:00. At eight o’clock Monday morning my email is going out. I am demanding that we deliver evidence-based care per this guideline at 8:00 on Monday morning.”

So that’s an example of a demand. Guess what, it really wasn’t all that effective. It also highlights the limitations of a single discreet strategy — a lot more is needed. But that’s an example of a demand, and I know you probably work in settings too where there’s a demand that we’re going to start doing it this way.

Sometimes that comes from policy. I’m really intrigued in working with some colleagues nationally on the fact that federally qualified health centers are going to have to start integrating behavioral health services in primary care with the Affordable Care Act. So there is a policy demand.

Sometimes there’s a push-out of a particular evidence-based practice. In mental health and social work we have several areas where people are developing treatments. Greg named some of them, KEEP, SafeCare, we have programs of treatment development in all of our fields. This is really, really important. And people who develop these interventions become people who work for very good reasons to try to push out and make available, and get adopted, and get scale up and spread of that intervention.

There’s sometimes a pull from the ground up, sometimes patients, family members, advocacy groups, sometimes through lawsuits, sometimes through — in child welfare, we have often our feet held to the fire by the press who uncover egregious events, so there’s a demand.

So assess what’s going on in the context to see what’s happening with implementation.

I can skip over the CFIR framework and help you find some resources for measuring. This again is the consolidated framework for implementation, the CFIR.

So when we’re doing our contextual assessment, we need to figure out the extent to which the practice change that we are proposing, or that we are demanding, or that we’re studying, to what extent is that practice change aligned. These are some of the terms that you’ll find in the literature to help you get into this.

A recent Flottorp article in the journal, “Implementation Science,” which by the way, is an online open access journal. You can go to the website; all the articles are there. It’s a terrific resource. It’s very searchable. So Flottorp has a checklist approach for what he calls “practice determinants.”

My colleague, Ramesh Raghavan at the Brown School at Washington University, talks about the policy ecology, we’re aware of what an ecology is. And he says, “Let’s apply that concept to practice.”

And Karen Emmons talks about system antecedents, and agency or setting infrastructure. So in other words, we know we can’t just airdrop a new practice into the real world without thinking of how it fits; is it aligned with, is it counter to what’s going on in the practice context?

So I hope those terms will help you do some searches to get at some measurement of this really important area.

Key Construct: Implementation Outcomes

The second construct I want to talk about is implementation outcomes.

It’s very important that we distinguish those from clinical outcomes, because we can have a very effective treatment, and if it’s poorly implemented, we are not going to see the patient or client changes that are hypothesized. And if we skip over those implementation outcomes, we might say, “Oops, didn’t work.” We might say, “It didn’t work,” when in fact it was never adopted, it was never delivered with some granularity that these implementation outcomes will help us capture.

And then of course we have ineffective treatments that are very successfully implemented all the time. So these are different concepts.

So some of the major categories of implementation outcomes; a beginning list is listed here. Colleagues and I put out a paper in 2011 that we hoped would jumpstart the field to begin to pay attention to this notion, develop some measurement approaches, and advance our list of beginning implementation outcomes.

Acceptability — to what extent is evidence-based practice acceptable to providers? What are their attitudes toward that? And I’ve been in situations where there’s been real pushback from providers, “I don’t like this.” There are parts of treatment protocols, for instance, an exposure treatment, or parent training, that go against the grain, and people don’t like to do that. Reliving some trauma exposure is a very difficult thing, and so we see a lot of clinician pushback saying, “I don’t like that. I don’t like that.” Well, if a particular program or treatment is not acceptable, you’re not going to get very far with the implementation.

Adoption is really the extent to which people do it. In our paper we distinguish this a little bit further — some of these are attitude, some of these are behavior, some of them refer to providers, so I refer to that paper for more detail. But do people really start delivering it?

Without procedure codes, many of us in our fields are left at this point with asking people, “Are you delivering evidence-based practice?” And guess what, they’re going to say yes. Or if we say, “Are you delivering this,” of course they are. But when you begin to look under the hood, you see that in fact it may not be quite the case.

We talked yesterday about feasibility; that many new treatments and programs are developed without having been really designed for dissemination, or designed for implementation. Brian Mittman talked about this a little bit yesterday. If they’re clunky, if they’re the Cadillac model, if they take thousands and thousands of dollars in training, and coaching, they may not be very feasible for certain practice settings. So this, again, is going to be a roadblock to implementation.

Fidelity: I think fidelity measurement is probably a big deal in your field, as it is in mental health.

Implementation cost is something that’s really not looked at as much as some of these other constructs. My colleague, Ramesh Raghavan is trying to develop procedures for costing what it takes to start a new program. First of all you have to take people offline. They’re out of clinical care so there’s lost billable hours. There may be training costs. Many programs and treatments have specific outcome monitoring requirements, so there may be costs associated with that. And then there’s generally the cost of retooling. Whenever we start doing something new, we have to allow some time, some energy, some turnaround that we don’t just keep perking along.

Penetration is a concept that really gets at the nuances of adoption. Penetration refers to the extent to which a new evidence-based practice is really filtering through and down into the system.

So let’s take, for example, a setting where there may be 20 therapists. And say that’s in one organization or one clinic. If you call them up and say, “Are you delivering this evidence-based treatment?” the answer may be, “Yes.” So when we begin to unpack that, how many of those 20 therapists are delivering the evidence-based practice? That’s another level of analysis.

Yet another one is within their own caseload, what proportion of clients who could benefit from this evidence-based practice are receiving it? So akin to the Shortell levels of change in a slide that Greg had up, penetration means that we have to take a more granular look, a more specific look, at the delivery of an evidence-based practice throughout the levels of a service organization.

And then finally sustainability is a challenging outcome, and one that David Chambers is going to talk in greater depth around.

So what we do know about measurement of implementation outcomes?

Well, first of all we know that fidelity is one of the most frequently measured implementation outcomes. It’s something that we know how to capture. When we think of stakeholders, treatment developers and researchers are really invested in that. They want to know if something is being delivered with the optimal fidelity. So there’s been a lot of attention focused on that.

We also know that provider attitude towards evidence-based practice have been pretty frequently assessed.

We also know that implementation outcomes are interactive, and that what happens in one of those outcome domains is really related to other implementation outcomes.

For example, I was involved in implementing collaborative care for depression among depressed older adults who were on Medicaid, with functional impairment, and high medical comorbidity. And the case managers in the Division of Aging said, “We know our clients are depressed, but who wouldn’t be? And their depression is pretty intractable. Nothing is going to help.” Well, when we took problem-solving therapy, did some behavior activation, guess what, the clients really did respond. Yes, they were still poor and still sick, but they were not as unhappy every moment of their waking day. So when the case managers saw that this treatment could be effective, well they were much more excited about it and they were much more accepting of bringing that program into the organization.

We also know the high cost reduces the perceived feasibility of implementing something new. And my conversations with agency executives say when they think about adopting something new, they say, “How much is it going to cost, and how much havoc is it going to wreck with my service delivery system?” So these implementation outcomes need to be measured. We need to take the temperature of where the setting, the stakeholders are, by measuring these, and then that can inform where we direct those implementation strategies, and how we begin to think of phases and implementation processes.

There are some developments; help is on the way, as they say, around measuring implementation outcomes.

There is what started as a Seattle Implementation Research Conference, or SIRC, funded by an NIMH grant. A group has been working hard to catalog and put online a repository of measures around both our 2011 framework and the CFIR.

And the National Cancer Institute jumpstarted a repository for implementation outcomes called “The Grid-Enabled Measures,” or the “GEM,”. I would encourage you to check either of these two websites for measures around this important construct of implementation outcomes.

Key Construct: Implementation Strategies

The last construct that I want to focus on is implementation strategies.

Implementation strategies are really the “how to” component of changing healthcare practice. I love Carolyn Clancy’s, the former executive director of AHRQ, characterization of implementation research. She said, “Implementation research is how to make the right thing to do the easy thing to do.” And we know that often the right thing is often not easy. So that’s our job as implementation researchers to figure out how to make it easier, if not easy. So it’s really the key to how to make the right thing to do easier.

Greg already provided the definition.

And he talked about the fact that the strategies vary in complexity. I think it’s good to take you back to this distinction, however, because for instance, my example of the email edict to implement the new guideline is a very discreet strategy. Greg’s illustration of the ARC model, with 12 steps, 4 phases, 3 major levels of change, that’s an example of a very complex strategy.

So, this citation provides a little bit more detail about some of the major types of strategies.

There are a lot of planned strategies which involve gathering information, building buy-in, initiate leadership, and we’re all eagerly awaiting Greg’s evidence tested leadership development models, and developing relationships. And I think woe to any implementer who skips over some of these planning strategies.

We’re very familiar with educational strategies. We’re here, right, in a conference. We do in-service training. We provide CEU’s. We develop materials. We develop manuals, protocols. So these are really our traditional ways of implementing new practices.

We’re very familiar with educational strategies. We’re here, right, in a conference. We do in-service training. We provide CEU’s. We develop materials. We develop manuals, protocols. So these are really our traditional ways of implementing new practices.

We also see a lot of action around restructuring. Roles sometimes have to get revised; new teams. Sometimes the service delivery site is changed. For instance, I referenced one already with the Affordable Care Act’s mandate that federally qualified health centers begin to deliver integrated mental health services, and behavioral health services within primary care. So that’s a prime example of restructuring federally qualified health centers, restructuring primary care. And guess what, the specialty mental health providers are not all that excited about it so they’re getting shook up too. So when we restructure roles, that is sometimes a very powerful and often effective, and in many cases necessary way to implement something new. That usually means that some record systems have to be changed, and that communication protocols needs to be changed.

There are financial strategies. Social workers wish there were more financial strategies in our field. We wish somebody would pay for services to vulnerable people who need case management, community organizing, and all of that. But particularly in healthcare we see a lot of great examples of strategies that involve pay for performance, incentive-based approaches. And in mental health care we are seeing some big change with parity where if mental health — behavioral health services begin to get paid for, we’re hoping that will facilitate the implementation of evidence-based care.

Quality management strategies; we have a lot to learn from the field of healthcare in quality management, where audit and feedback. We have a lot of evidence that simply counting what happens and telling you how you behave changes your behavior. So audit and feedback is very, very helpful. Clinician reminders; those can take the form of pop-ups in electronic records, or sometimes if there are not electronic records, people are using cards, or checklists, which a provider can use. Technical assistance, cyclical tests have changed.

And then there are policy strategies which involve licensure, accreditation, certification, and liability. We’ve seen a lot of cases in mental health where certain kinds of care have been legislated or mandated, often out of court, class action suits.

So what do we know about the effectiveness of these strategies?

Well, first of all we know that passive strategies are really not very effective. Reading an article — who knew that if people read our articles they just wouldn’t go out and start behaving differently. And issuing memos.

Training is the most frequently used strategy, but we’re learning that multi- component strategies are most important.

I don’t know about you, but I don’t really like having somebody tell me my behavior needs to change. And I think if we think about providers in that regard, who would like to have somebody come in and say, “Oh, please, start doing it this way”? Usually our behavior works, right? It’s worked so far so why change it?

So to change we’ve got to tackle those forces of inertia, those positive payoffs that we’re getting, and a lot of stuff has to be shifted. And that I think is kind of an individual window into why multi-component and multi-level strategies are most important.

Now, I have a beef with the current literature about implementation strategies. And that is that it doesn’t give us as much detail as we need. Things are named, things are tested, but they don’t tell you who does what, when, where, for how long and how much. And just as we know that treatments have to be specified, and need protocols, and manuals, and measurement, and dosage, we have to begin to have these same expectations and needs for implementation strategies.

So our team just published a paper calling for better reporting and implementation strategy articles specifying the actor, the action, the timing, the dosage, and the target: why were you doing restructuring, what part of the context, or which level of service delivery were you targeting? This paper calls for more specification, more reporting of implementation strategies, akin to what we are now used to expecting with treatments.

And Brian was with us yesterday, but he’s given a couple talks at our Implementation Research Institute where he says that implementation strategies should be theory-based, presented with a logic model. We need to begin specifying the mechanisms of change for implementation strategies, multifaceted and multilevel if appropriate, feasible and acceptable, compelling, observable, sustainable, scalable. So these are some of the additional characteristics that we need to begin expecting of implementation strategies.

Data Sources

So I’m going to close with just a few observations about the kind of data that are used in implementation science, the kind of data you can begin to identify and try to use.

There are a number of participants in implementation. So on the left-hand side here we have — these are some of the participants, and these are some of the data sources.

For administrators we often survey them, or ask them through key informant interviews; similarly with supervisors, front line providers. Support staff we see a lot of focus groups about what’s going on, and service users. And there is an interesting approach called group model building, which is really a quantitative structured approach to focus groups where we illicit the input and the suggestions and kind of run out simulations of what — if we did that, what would happen? And what would be — yesterday we were reminded to think of the negative things, what can go wrong, because some of it probably is going wrong, adverse events. So group model building can help us lay all that out and get a picture of how we might begin to better understand the complexities of implementation.

Implementation processes are often studied through ethnographic observation. People can go in and hang out, they can use these qualitative approaches. Greg said that he became a convert to more mixed methods in qualitative research when he realized that a lot of the action on the ground, a lot of what’s happening in the context is not as readily captured through administrative records, or readily available quantitative data.

If we’re looking for the footprint, or the impact of implementation, in other words, did practice really change, then we usually have to go into document review, is there evidence that less of something was used, and more of something new began being used; charts, records, board notes, you know, did an executive director take a decision that we’re going to go down this new path to a board, and budget line items. Many people say, “If it’s not being paid for, it’s probably not being implemented.” So if we can begin to look at agency or provider budgets, we may get a sense of what’s going on.

So these are just some of the potential data sources; again, with qualitative and quantitative data.

References

Chambers, D. A., Glasgow, R. E. & Stange, K. C. (2013). The dynamic sustainability framework: Addressing the paradox of sustainment amid ongoing change. Implementation Science, 8(1), 117 [Article] [PubMed]

Damschroder, L. J. Aron, D. C. Keith, R. E. Kirsh, S. R. Alexander, J. A. Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science.Implementation Science, 4(1), 50 [Article] [PubMed]

Emmons, K. M., Weiner, B., Fernandez, M. E. & Tu, S. (2012). Systems antecedents for dissemination and implementation a review and analysis of measures. Health Education & Behavior, 39(1), 87–105 [Article]

Flottorp, S. A., Oxman, A. D., Krause, J., Musila, N. R., Wensing, M., Godycki-Cwirko, M., Baker, R. & Eccles, M. P. (2013). A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implementation Science, 8(1), 35 [Article] [PubMed]

Grimshaw, J. M., Shirran, L., Thomas, R., Mowatt, G., Fraser, C., Bero, L., Grilli, R., Harvey, E., Oxman, A. & O’Brien, M. A. (2001). Changing provider behavior: An overview of systematic reviews of interventions. Medical Care, 8(Suppl 2), II2–II45

Grol, R. & Grimshaw, J. (2003). From best evidence to best practice: Effective implementation of change in patients’ care. The Lancet, 362(9391), 1225–1230[Article]

Powell, B. J., Mcmillen, J. C., Proctor, E. K., Carpenter, C. R., Griffey, R. T., Bunger, A. C., Glass, J. E. & York, J. L. (2012). A compilation of strategies for implementing clinical innovations in health and mental health. Medical Care Research and Review, 69(2), 123–157 [Article] [PubMed]

Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C. & Mittman, B. (2009). Implementation research in mental health services: An emerging science with conceptual, methodological, and training challenges.Administration and Policy in Mental Health and Mental Health Services Research, 36(1), 24–34 [Article] [PubMed]

Proctor, E. K., Powell, B. J. & Mcmillen, J. C. (2013). Implementation strategies: Recommendations for specifying and reporting. Implementation Science, 8, 139[Article] [PubMed]

Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R. & Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda.Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76 [Article] [PubMed]

Raghavan, R., Bright, C. L. & Shadoin, A. L. (2008). Toward a policy ecology of implementation of evidence-based practices in public mental health settings.Implementation Science, 3(1), 26 [Article] [PubMed]

Tabak, R. G., Khoong, E. C., Chambers, D. A. & Brownson, R. C. (2012). Bridging research and practice: Models for dissemination and implementation research. American Journal of Preventive Medicine, 43(3), 337–350 [Article][PubMed]

Further Reading: Websites of Interest

CFIR Research Team, Center for Clinical Management Research . (2015). CFIR Technical Assistance Website. cfirguide.org (Referenced as CFIR WIKI http://wiki.cfirwiki.net/ in the presenter’s full PDF slides. Note the change in name and location of this resource).

National Cancer Institute. (2012). Grid-Enabled Measures (GEM) . Behavioral Research: Cancer Control and Population Sciences (Available from the NCI website at cancercontrol.cancer.gov).

Society for Implementation Research Collaborative (SIRC). (2015). The SIRC Instrument Review Project (IRP): A Systematic Review and Synthesis of Implementation Science Instruments. SIRC: Initiatives (Referenced as Seattle Implementation Research Conference Measures Project / www.seattleimplementation.org/sircprojects/sirc-measures-project in the presenter’s full PDF slides. Note the name change of this organization).