Right Care, Right Patient, Right Time: The Role of Comparative Effectiveness Research

Please note: This is an unedited transcript. Note: This is an unedited transcript. For direct quotes, please see video at: http://allh.us/GNv4


SARAH DASH:   Good afternoon everybody.  Thank you so much for being here on this beautiful spring day.  We are delighted here at the Alliance to be hosting a briefing today to talk about Right Care, Right Patient, Right Time and the Role of Comparative Effectiveness and Patient-Centered Outcomes Research in helping to deliver the care that best aligns with patient’s values and preferences and clinical outcomes today.


So my name is Sarah Dash, I’m President and CEO of the Alliance for Health Policy.  How many people have been to an Alliance briefing before?  If I could just get — oh, you guys are great.  Thank you.  Thanks for coming back.  If you don’t know about us, we are a non-partisan, not-for-profit, really dedicated to advancing knowledge and understanding of all types of health policy issues and we are here as a resource for you all, so we are really glad you’re hear.


Before we get started, I would like to thank the Patient-Centered Outcomes Research Institute, PCORI, for their support of today’s briefing through a Eugene Washington PCORI Engagement Award.  You can follow along today’s discussion, if you like, on Twitter at the hashtag #allhealthlive and you can submit questions via Twitter as well.


We have an amazing panel today and they are going to tell us about how we use evidence to improve care, to — as I said — identify which treatments, therapies, modalities, may work better for different patients, and how we are learning about incorporating patient viewpoints and preferences into really kind of the whole continuum of care delivery.  We are going to explore the purpose and perspectives surrounding comparative effectiveness research, CER, patient-centered outcomes research, or PCOR, and other types of research.


So I’m going to go ahead and introduce our panelists without further ado.  You do have their full bios in your packets and then we’ll go ahead and get started.  So first, joining us today, all the way at the end of the table from me, is Eleanor Perfetto.  Eleanor is Executive Vice President of Strategic Initiatives at the National Health Council, where she conducts research and policy work on patient engagement and healthcare, including comparative effectiveness, patient-centered outcomes research, medical product development, value assessment, and healthcare quality.  We’re really grateful to have her here today to discuss why CER is important to patient groups and what they are looking for in research results and policies.


Next, we will hear from Gail Wilensky.  Gail is a Senior Fellow at Project Hope and should say, former administrator of the Healthcare Finance Administration under the first President Bush.  Her expertise is on strategies to reform healthcare in the United States with particular emphasis on Medicare, comparative effectiveness research, and military healthcare.  And she is going to tell us about the historical context of CER and how that brings us to today.


After Gail, we will hear from Sean Tunis, Founder and Senior Strategic Advisor at the Center for Medical Technology Policy.  Sean works to provide a neutral platform for multi-stakeholder collaborations that are focusing on improving the quality relevance, and efficiency of clinical research.  He will use his time to share how the culture of clinical research has shifted, to emphasize evidence and the patient voice, as well as to describe various methodologies.


We’ll then hear from Dr. Alfiee Breland-Noble, also known as Dr. Alfiee to her patients.  She’s Project Director of the African-American Knowledge Optimized for Mindfully Healthy Adolescence, housed within the Center for Trauma in the Community at Georgetown Medical Center.  So her research focuses on reducing mental health disparities and racially diverse adolescent youth and families and she’s going to discuss how a CER study is conducted, including how patients, especially those who are typically excluded from clinical research and trials are involved — or can be involved throughout the process.


Then finally we’re delighted to have with us John Bulger — Dr. John Bulger is Chief Medical Officer at Geisinger Health Plan.  He will explain how the health plan uses comparative effectiveness in practice with their patients and tell us about Geisinger’s experience with the proven care hepatitis C program as a case study.


So with that, thank you all so much for being here.  I’m going to turn it over to Gail to kick things off and I’m going to just hand this down — I’m sorry, to Eleanor to kick things off — I apologize, and that’s — just give that to Gail for when it’s her turn.  Let’s go.


ELEANOR PERFETTO:  Thank you.  I don’t have any slides, thanks.  So my job here today is to kind of kick things off with grounding everyone in the room with some background and some definitions.  And so the first thing I want to start off with is telling you a little bit about what the National Health Council is.  We are a not-for-profit membership organization here in Washington, D.C., and our membership is predominantly made up of patient advocacy organizations.  We have other organizations in membership such as not-for-profit groups that are interested in healthcare like faster cures for example.  We also have professional associations, we also have business and industry, so — pharmaceutical companies, health insurance companies.  But the largest group that we have in membership are the patient advocacy groups and I know that some of them are here today.


So we focus really on what’s important to patients through our patient advocacy group membership and they really drive the work that we do and our mission is to be supportive of them.  So to give some context for — we’re talking about comparative effectiveness research in patient-centered outcomes research and we’re really talking about two sides of the same coin.  They are different, but they are very related to one another.  So I wanted to start off talking about that, because we have had this evolution from what originally started off as conversations about comparative effectiveness research and have now evolved into comparative effectiveness research that’s really patient-centered and patient focused research.


So what is comparative effectiveness research?  Well, a very formal definition is that it is research that compares two or more things to one another to kind of figure out which one works best.  Only it has to be much more nuanced than that.  You know, we had some old conversations about which works best — the pink pill or the blue pill?  And we know that this is much more complicated question than which one of them works best.  It’s actually, which one of them works best for which patients, because sometimes the pink pill will work for some patients and not for others, and sometimes the blue pill will work for some patients and not for others.  So comparative effectiveness research is really focused on, can we figure out and can we drill down and understand while we avoid doing trial and error with patients, can we figure out which one will work best for which patients, and make our selections that way so that patients can be part of the conversation and their clinicians can be part of the conversation, knowing full well which one they expect to see work best for them.


The other side of that coin is patient-centered outcomes research, and that’s making sure that we have patients become more involved in the research process.  Not as study subjects.  We will always need to have patients be study subjects, but rather as partners in the research themselves.  And that really changes the way that we do research.  So it really gets to why patient-centered outcomes research is so important.  We had research that was predominantly dominated by the scientists and the researchers who were planning the protocols, and the questions were driven by those researchers.  The outcomes they were choosing were driven by those researchers.  And we had designs for the research that might have been pristine designs really getting to a very elegant research study, but in the end it wasn’t answering the questions that patients and families needed to be able to have to make the decisions for themselves.  And so when we have — when we talk about patient-centered outcomes research, we talk about questions that are important to patients.  It’s the questions that they need answers to, that will help them make choices.  We talk about the outcomes that are important to patients.


So when we look at the end points for those studies, are we zeroing in on the end points that patients really care about as opposed to the end points that the clinician or researcher might care about?  And then also patients being involved in informing the design of the study, so that there will actually be studies that patients will participate in.  Now, we want to avoid having too many drop-outs in studies, or a protocol that someone won’t sign up for because it sounds too onerous to them.  So that’s really what getting at patient-centered outcomes research — to get at comparative information, is all about.


So in summary, I want to say that what patients are really looking for is information to help them figure out what’s right for me, or for a family member when they’re making that decision.  They want to be able to sit down with their doctor and say, of the options I have, which one will work best for someone who looks like me, and for someone who lives in my circumstances and has the preferences that I have?  And I think we have to kind of stop now and think:  Are we there yet?  Because we’ve seen this big transition to go from not having patients engaged, now having patient centered outcomes research, but in the last ten years we’ve seen it happen.  We’ve seen that transition happen.  And with PCORI really spear-heading and leading a lot of this work, we’ve seen that transition come to the place where even organizations like the Food and Drug Administration are looking at patient-focused drug development, and really have taken leadership there to move this forward within the clinical research realm of new medical product development.  And so when you see those kinds of changes happening in the research culture, you know that we’ve actually seen quite a shift in the way that we do healthcare research.  I think one of the things that we have to all accept is that our work is not finished, because there’s so much work to be done.   And if we started off thinking ten years ago that we were going to be done in ten years with having all of this work done, we were quite naïve if we thought that.  We should have always known that this was going to take longer to do, because there is so many patient populations, so many diseases, so many very specific questions that we need to get the answers to.


But right now, I think one of the important things to take away is that patient-centered outcomes research and comparative effectiveness research are expected by patient populations because they still need that information in their decision making.


SARAH DASH:   Thank you so much, Eleanor.  Gail?


GAIL WILENSKY:   Thank you.  Pleasure to be here.  I wanted to put some of today’s discussion in a bit of a historical context.  Both how in the United States we began to get interested in the concept and how personally as a market-oriented economist, I began to become personally invested in the concept of comparative effectiveness research.  And it really had to do with the notion of how to enable both patients and physicians to be able to be in a position to make better decisions.


My own position on this came about realizing that we had — and have had over several decades, several challenges in healthcare.  One is that we’ve had spending growth rates that are unsustainable.  And while it slowed for a few years, it is still expected to grow at a faster rate than the economy.  And in addition to that, we have had both problems with patient safety and problems with quality.  So spending a lot and not always spending it as effectively as we might.  When you think about comparative effectiveness information, it’s important to think about it not as an end point, but rather as a basic building block, providing information on what works when for whom, provided by a particular type or place of healthcare provider.  This was a point that Eleanor raised.  It’s not an answer that is likely to be true for all people at all times and all places.  And also recognizing that technologies rarely are either always effective or never effective and it’s trying to understand again, for whom, when, under what circumstances, particular technologies might be effective.


If we look at where other countries have been on this process and frankly they have adopted it at an earlier point, we see that it is a very centralized process in most countries.  Literature review focus, or actually looking at experimental designs.  Usually by agencies that are part of the government, which is not surprising, since in most of these countries the government is a major part of the payer, if not a completely centralized system.  Whether or not the recommendations are mandatory, or the transparency of the process differs.  It was clear to me, in thinking about it, that the U.S. needed something different.  We need to figure out how to spend smarter.  If you’re not going to use direct regulatory control, you both need better information and ultimately better incentives.  And it meant focusing on conditions, rather than as they did in Europe, frequently focusing on a particular therapeutic or intervention.  And not just drugs and devices, but surgical procedures versus medical procedures.  Invest in what is not yet known and use what is known more effectively.  Recognizing it’s a very dynamic process.


The place to start has always seemed to me to look at the high-cost medical condition, where there is a lot of variation in treatment.  Conditions that are reflecting the highest class DRGs with substantial geographic variation, for example, is a good proxy.  And we also, because it’s so much to encourage that private funding goes on, but subject to agreed upon guidelines with results that can be audited, only because we need to recognizes that it’s very unlikely that the government alone, through PCORI or any other agency, is likely to be able to get all of the work done that needs to happen.


We also need to recognize that if we’re going to get good decisions, we are going to need to have data from very different sources.  We sometimes tend to think about the double-blinded, randomized control trial as the gold standard.  Actually, what it is, is the way to get rid of selection bias.  But it frequently introduces other bias or other deviations from what is likely to occur when something is actually used, because different kinds of providers or institutions are providing it, or because the patients are different from those that are in the RCTs.  Sean, to my left, has talked about real-world RCTs and I hope he’s continuing to talk about that.  Using epidemiological studies, administrative data, it’s important to understand that all data have limitations.  It was an invaluable lesson I learned as the co-director of the first of the big expenditure surveys, the National Medical Care Expenditure Survey, now called the Medical Expenditure Panel, is that all data has errors and all data has limitations and that we need to look at the various data sources in order to learn what we can learn from them.


But as important as it is to get good information about comparative effectiveness research and unfortunately we are still very much in the early stages of both the investment and of actually learning what it means for different classes or groups of individuals, they have particular types of interventions.  I’m an economist and so for me this is an effective building block to get information about what works best for whom, or what works better for whom under various circumstances.  It’s not the end of the decision making.  If we’re going to use this as a building block to get better outcomes, which I hope that we’ll continue to do, we also need to have better incentives in place.  We need to make sure that the financial incentives between the people providing the service and the technology and the people using the service are in sync, otherwise you’re not going to get a good outcome.  We need to make sure that we’re rewarding institutions and clinicians who provide high quality and efficiently produced care.  We talk a good game, we actually rarely do that very well.  We’ve talked about using value-based insurance in private sector. It is more absent than present, or it affects a very small part of the service that is delivered.  And ultimately we also need to understand that if we’re really going to get better at healthy outcomes for individuals, we are going to need to encourage them to be a part of the process also.  Encourage them, reward them for healthy lifestyles and maybe discourage them from other lifestyles.  It’s a critical component to have better information, but it’s not the only thing we’ll need to have the outcome we want.  Thank you.


SEAN TUNIS:  So good afternoon everybody.  I wanted to thank Sarah and the folks at the Alliance for inviting here.  Also wanted to thank the PCORI folks who are here, and all the folks at PCORI who for the last eight years I think have done a tremendous job of kind of advancing the field of patient centered or comparative effectiveness research.  You know, back when I think Gail started writing about this in 2006, the ideas were out there.  I think — or thanks to Gail and some others, you know, kind of pulling this off and creating a real science and discipline out of a field that didn’t exist before.  It’s been great to watch it happen and I think we always thought from the beginning that this was sort of a generational — you know, it was a generational agenda, not something — not an issue that was going to be solved in five or ten years.  So you know, I think things are on a good track.


I’m going to do a little bit of looking backwards and then looking forwards around comparative effectiveness research.  I think some things that have happened are really strong and powerful and moving the field forward, and there are some things that might be useful to emphasize more going forwards.  So back in the early 2000’s, I was working at the CMS.  I guess, Gail, when  you were there it was called HICVA.  Sometime after this, if you want to hear the stories about how Tom Scully decided on the name CMS, I’m happy to share those with you.  But the whole — so one of the functions that I oversaw at CMS was the coverage decision making and deciding which new technologies to pay for, and as part of that process we would ask the AHRQ — Agency for Healthcare Research and Quality — to do these hundreds of pages of reviews of all of the available evidence and what I’ve came to call the “evidence paradox” was this kind of what — this recurring problem that 19,000 randomized trials are published every year, tens of thousands of other clinical studies, and virtually every report that we got from AHRQ, not matter what the topic, concluded that the evidence is inadequate or poor quality.  And you’d think with 19,000 shots on goal, you know, every once in a while.  So this is a puzzling phenomenon.


So when you started to kind of pour down deeper into what the problems were with how research was being done, some of the issues were — these are a little miss formatted, but that’s probably my fault.  First of all, that the research agenda was being driven by clinical investigators, researchers, and not really informed by the target decision makers — patients, clinicians, payers, were very — had very limited involvement in deciding which research questions were going to get looked at.  Part of the problem just derived from who got this thing going.  Then typical studies were limited to narrow patient populations, they were often done in academic centers, not in contexts that were generalizable, and often times studies were designed comparing to placebo or treatments that were not in wide use.  So a big part of the sort of underlying issue was the research agenda just wasn’t aligned with what the real questions were that people who had to make healthcare decisions had to make.  So that was kind of the — that then, I think, is what primarily fed all of these gaps in evidence and the ability of anybody to make well informed decisions.


A way that I’ve come to sort of — underscoring this, is using — and finally, the most important thing and why I think the name “patient-centered outcomes research” is so critical, is it’s about making sure you’re actually measuring the outcomes that matter most to patients, not the outcomes that are the easiest to measure, or that are most reliably measured.  So this was a big issue.


It’s not important to read the details on this — but does anybody recognize what this comes from?


SPEAKER:   Consumer Reports.


SEAN TUNIS:  Yeah, it’s a page from Consumer Reports.  It happens to be a Consumer Reports electric screwdrivers.  The reason I’m using this is to see — you know, on the left is all the different — looks like Japanese electric screwdrivers — and then across the top you have all of the features that consumers care most about when they’re buying electric screwdrivers — like the speed and the power, the charge time, the run time, all of those things.  Well, if you think about it, all of those features that consumers care about are determined through careful focus groups and interviews.  They say, well, you know, what does a consumer care most about?  And then those are the things then that the engineers go and develop bench tests so that you can directly compare the different brands on those outcomes that matter to people who buy the service.  Believe it or not, this doesn’t happen in healthcare, or hasn’t happened, right?


So this would be a consumer reports table that you would find if you were looking at a drug.  Right?  It’s like, for some of them they measure the power and the charge time, but the studies don’t measure the run time, et cetera, et cetera.  So if Consumer Reports filled their pages with tables like this, nobody would buy the magazine.  You can’t possibly be an informed consumer.  And so I think one of the things that patient-centered outcomes research got right was, there’s no way you’re going to produce useful information unless you first concentrate on what actually matters to the people that are going to be using the services and apparently that was obvious for electric screwdrivers, but not obvious for medical devices or drugs or diagnostics, et cetera.


It looks like I’m only going to have time to do a little bit of kind of a political history that I think is relevant to today.


SARAH DASH:   You can take a little extra time, because that’s important.


SEAN TUNIS:  Okay.  So there’s people in the room who will clearly want to correct me, I hope they will stay quiet — no.  So the original draft legislation to create an institute to do comparative effectiveness, it was going to be called the Comparative Effectiveness Research Institute; that became a problem in the context of the Affordable Care Act debate, because that institute became thought of as the “Death Panel” that was going to determine whether old people were well enough to get a hip replacement, right?  So the institute that was planned became sort of synonymous with the death panel.  So the clever people, I think I’m pretty sure it was in the Senate, on the finance committee, decided to rename the Comparative Effectiveness Research Institute to the Patient-Centered Outcomes Research Institute, because who could be against patient-centeredness, right?


Well, once the name was changed and through a lot of effort to kind of get this — all of the language right et cetera, the PCORI leadership, once the institute existed, took the name seriously.  So they decided well, this is going to be a patient-centered outcomes research institute and what’s more, they decided cleverly, that to avoid any of that “death panel” spillover, they are going to avoid talking about costs or cost effectiveness or qualities or even payers.  But the problem with that is, as Gail said, one of the original inspirations for setting this thing up was to help people make cost-conscious comparative decisions in the face of unsustainable spending trends.  You know, it sort of narrowed the space within which I think PCORI felt comfortable initially operating.  Also, as a result of that, when the agency, I think, institute took seriously the idea of being patient-centered, they put a lot of energy into talking to patients and talking to patient groups and finding out, well, what were their important questions?  And it turns out, they had a lot of questions about things like care coordination and sort of the, how could they have a better care experience and weren’t so interested in those kind of Consumer Reports type questions — not that they were disinterested, but those didn’t rise to the surface.


Then sort of a last point — I won’t go over too much because we can come back to things in the Q&A, but what I think of the many positive unintended consequences of the name change was this very dramatic and important focus on what patients really needed to know and as a result — and I think Eleanor, you said this too, as a result of this sort of ah-ha moment that — taking into account the perspectives of patients actually would help you design more relevant studies in ways that were more informative to clinical decision making.  You know, that sort of percolated out and it influenced the National Institutes of Health, which started a whole pragmatic trials collaboratory to do, and much greater engagement of patients.  The FDA’s patient-focused drug development I think kind of exploded and expanded because of the work being done at PCORI.  And then I can tell you, because I work a lot with the Pharma device diagnostics industry, they’ve really gotten the message that they need to engage patients from phase one and phase two studies, and they are making decisions about which products to actually develop based on early feedback and input that they’re getting from the patient community.  That never happened before, and I think it’s going to mean that the products that start to appear now and in the future are really going to be those that have more meaningful benefits to patients and are actually studied in ways that will be much more helpful for patients and clinicians to be able to make informed decisions.


Sorry to have gone over, but thanks!


SARAH DASH:  Great.  Thank you, Sean.  So we’ll hear from Dr. Alfiee next.  And we’re going to come back to some of these points.


ALFIEE BRELAND-NOBLE:  Okay.  Good afternoon.  You are all so quiet.  Okay, I’m used to standing up, this is a little different for me.  So I want to resist the urge, but I can’t help myself, so I have to say it because she’s one of my idols.  My lovely colleague, Dr. Tunis has gone over just a little teeny — which is fine, but I’m going to be like Auntie Maxine and I’m going to reclaim my time.  But you were wonderful, I learned a lot.  Thank you so much.  All right, so I’m looking at the time, it’s ticking, and I like to talk so I’m going to try to get through this.


I’m going to say, good afternoon, and I’m just going to assume you’re all sending back good wishes because you are all so quiet.  So we will just go with that.  My name is Alfiee Breland-Noble and let me tell you real quick — in communities of color, I see a lot of diversity in there, we like titles, okay?  Because sometimes the title as a person of color — I’m clearly an African-American woman — the title confers the kind of respect that sometimes you don’t get in other settings.  So when we say Dr. Alfiee, it’s not because I want to dismiss all of the wonderful people up here on the stage with me, it’s just because that’s what we do in communities of color, right?  So I have to set that stage, because that’s a part of what we’re going to talk about with my portion of this work.  So as my colleague so wonderfully — she got through it — I was so proud every time somebody gets the acronym.  They get all the way through it and they don’t stumble, I want to jump up and cheer.  Because it took me a long time to put that acronym together many years ago.


So what we do with the AAKOMA project is we focus on the mental health needs.  We started with African-American youth as an underserved population.  Why?  Because they are significantly underserved in terms of the services that they get for mental illness, right?  They don’t get the same services as other folks.  We have expanded that to include other communities of color over the years, right?  So my one saying that I always share with people — I do a lot of public speaking and some TV work, “a rising tide lifts all boats”.  So if we can figure out how to reach the young people who are the least likely to be touched by good treatments — good treatments being the operative term — for depression particularly, anxiety and other mental illnesses more broadly, then we can teach everybody how to get good treatment, right?  And so when we look at what the disparities are, the disparities are not necessarily focused on the prevalence of mental illness, particularly depression, because when you look across racial ethic groups, those rates are about the same in terms of our epidemiological research.  Where you see the difference is in who gives care and what is the quality of care that they get.


So that’s what we’re about.  We want to educate people about these issues with the AAKOMA project in collaboration with community. Right?  So when we talk about patients, a lot of times people get the impression it’s us and them.  I’m a provider, many folks up here are providers or have been providers in some way, and those folks out there are patients.  We are all patients.  Right?  At some point in our lives, we are all patients.  So what I try to do is keep that at the forefront in my mind when I’m working with community members.  So I like to think in terms of community, which, PCORI, they know I love them, they have been very supportive of my work — we are called patients.  Right?  So we are thinking about community and patients, interchangeably.


This is just a picture that reflects some of the work that we do.  Almost all of our work is in collaboration with community.  We do a lot of community engaged work and a lot of community-based participatory research in the context of PCORI and CER.  A lot of alphabet soup.  So I share with you just a teeny bit.  Again, I’m reclaiming my time.  And I want to share with you that what are some of these barriers to care faced by young people of color?  So you see there are four listed here:  Decreased access to, less availability of good mental health services, lower likelihood of receiving required services.  The single biggest provider of mental health care for African-American youth based on the research is the juvenile justice system.  That’s when our kids get care.  Why?  In part because there are disparities in terms of you take the same constellation of symptoms like depression.  You have a white child, you have a black child or a Latin X child.  The white child is going to get mental healthcare, the black child and Latin X child are going to be diverted into juvenile justice, right?  So with the same set of symptoms.  That’s a problem, right?  Because you’re not getting the kind of care necessarily that you might like to get if what you’re diagnosed with is not depression, it’s a disruptive behavior problem.  You’re going to get different kinds of treatment.  Right?  So those have consequences.  For young people of color, African-American and other young people of color, they have experiences with poor quality of care and they are also under-represented — this is my biggie in terms of people who lead the research and in terms of our participation in research, right?


So that’s all the background.  When it comes to patient-centered research, part of what we want to try to do is reduce these barriers to care.  Because I’m limited on time, I won’t get into that in detail, but you have that in your packet.  So what do we do with AAKOMA — look at all of these acronyms — it’s a beautiful alphabet soup.  AAKOMA, CER and PCORI.  So what do we do?  Our objectives are for our participants to understand PCOR and CER so they can be informed consumers.  And informed participants, which is part of why I think I was asked to participate today. We want young people in particular — because that’s what I do, I work with young people.  I want young people to be clear about what are PCOR and CER so that they can be informed consumers and participants.  And we want to teach our communities effective ways to engage, and we’ll talk about two parts of that process.  One is an engagement award that we had through PCORI — both of these were funded by PCORI, and our goal was to build a curriculum to train black folks and black faith communities how to develop their own research — we call them ministries, because I happen to be like a — okay, you are all looking at me.  I’m like a lunar eclipse, you are only going to see one every 100 years.  I’m black Catholic, but I work with black churches — black Protestant churches.  And part of what we do in that context is we wanted to help build ministries because so much social justice — you all know this — this is the history lesson everybody gets, happens in black churches, right?  Think about civil rights movement, think about HIV/AIDS crisis, when it first started, early ‘80s.  Black churches were the places where blacks galvanized support.  And so we built these research ministries. We built a curriculum to teach people how to build these research ministries so that they could be effective consumers for folks like us.  So if I want to go out to work with a black faith community to do some kind of CER PCOR research, well now the community is informed about how I’m going to go about doing this work and they can grill me and ask questions.  There is nothing better than a well-educated consumer.


All right, and so what did we do?  The biggie on this slide, I want you to see 203 individuals.  You have to ask PCORI how much you get for engagement award.  I’m not complaining, I’m just making a point.  203 people with the amount of money that we had, is a lot.  So our primary outcome, we achieved it.  We built a curriculum, a built capacity in the community, we also were able to do very strong community and patient engagement.


So the last — I promise I’m only going to go about 30 seconds — part of what we also did was gain an understanding of what do black youth and adults know about PCOR and CER in the context of mental health, right?  Then we had a pipeline to proposal award — I’m going to skip through this quacking — part of what our goal was, was to develop in collaboration with community a PCOR and CER question and to put together a proposal.  These were the stages of the pipeline to proposal awards when we had it, and what did we end up with?  We ended up with building a trauma focused intervention.  That’s what the community wanted.  That’s not my expertise.  I’m a depression expert.  But the community wanted trauma, so we settled on a trauma informed style of care — electronic versus sort of in vivo care, focused on depression from trauma exposure.  It was a collaboration.  This is what our result was, we submitted a grant to PCORI last spring, we got a 41 — which was great, and we had full participation all throughout the process from the community partners, including the teenagers.  So you can see that score.  We had a good proposal with many minor weaknesses.  I disagree with that, but I’m not going to argue.  And so future research is to continue to build on this work in collaboration with community, with patients, with young people.  And I think that might be it.  And with that, I’m done.  Thank you so much.


SARAH DASH:  Thank you.  So I want to ask, before we move to the clinical setting, Dr. Bulger, I want to ask a follow-up question, if you don’t mind, to Dr. Alfiee.  What was the hardest part about engaging people in the community with the research project?


ALFIEE BRELAND-NOBLE:  I think for us it was — going back to something I heard a couple colleagues say — trying to find a balance between what are the questions that the researchers want answered — because that’s what you’ve got to convince — because many of the people who do research — who do review, I’m one of them, for PCORI, review those grants and it’s a very different set of metrics that they are using, I think.  So it’s creating a balance between that and what do patients care about.  Right?  So the patients may care about — I may care about a change in CER depression score — I’m sorry, CES.  The Centers for Epidemiologic Studies, depression scale.  I may want to see a drop in that scale.  The patients don’t care about a drop in that scale.  What the patient cares about is am I going to be able to get up and play with my kids in the morning if I have been suffering from depression for a long time and haven’t felt well enough to be able to do that?  So it really is striking a balance between what do scientists — including myself — what do scientists want versus what do patients want?  The other thing is building enthusiasm around the kinds of questions that we’re interested in.  Because I’m a mental health person, you know, mental health is super stigmatized in communities of color.  So trying to convince people that this is something that they should be concerned about, I think was also an issue, and I think where we found the leverage, was to appeal to people’s sense of caring about their kids.  If your kid had a broken leg, you would go see the physician so you can go get the broken leg fixed.  You want to think about depression in the same way.  So I think those are some of the hurdles that we struggled with that I feel like we were able to do a really good job.  You know, 203 folks talk about mental health issues — it’s a lot for folks in a black community.  So I would say I think that’s how I would answer that question.


SARAH DASH:   Thank you.  Dr. Bulger?


JOHN BULGER:   Thank you, it’s great to be here.  It was a nice drive down here this morning from North Central Pennsylvania.  I got lucky and hit 270 at the right time, so it wasn’t too bad getting down here.  So thank the Alliance and PCORI and you guys for spending your lunch with us today.


I’m going to talk about really — and I think the kind of real world RCT idea is a lot of the way we look at this and really what we engaged in was both a study of taking care of a disease — high impact disease as Gail talked about — but also of how do you work on delivery science?  How do you get to the point where you can take the research we’re talking about and actually put it into practice so it helps real people.  Many times we do a whole lot of research and we think we know the best way to do things, or we look at the best way to do things, but it doesn’t get put into practice and probably all of you have heard or seen a headline where it talks about the fact that we have the evidence, and we know what the right thing to do for people is, but it takes 17 years to get that to where the doctor who’s taking care of the patient, or the pharmacist, or the nurse practitioner, whoever the provider happens to be actually is able to do that.  So what we were looking at was partly how do you take care of that disease, but a lot of how do you get this so that the patients can get the right thing at the right time, at the right place, which was the title of this.  So we looked at — and what I was specifically going to talk about is Hepatitis C.  You’ve all heard about these new biologic medications that are out there that are very high impact, they are cures.  In the case of Hepatitis C, the drugs we have now actually cure the disease.  So you can get treated for 8 to 12 weeks and take a disease that was a chronic disease that ended up having a lifetime of care, sometimes transplant, sometimes other things, and taking a pill for 8 weeks, you’re cured.  Not just literally cured, actually cured of the disease.  And that has a big impact, when you think about it, on patients and families.  It also has a big impact on the healthcare system.


The other piece of this is though that those drugs don’t work for everybody, so you have to figure out the right people to get them to and as we said originally — and now PCORI doesn’t look at cost, but they are extremely expensive.  So four weeks of the drugs we’re talking about are relatively $30,000 to have the treatment for four weeks.  And generally you need 8 to 12 weeks to treatment.  Again, you get a cure, but having that money to care for that is a big part of the system and while you see this is our own data, Geisinger, but the bar graph is the percent of drugs that we prescribe.  So it’s less than one percent of the drugs are these biologic drugs.  But the redline is the cost.  So they make up about a third of the cost of all of the medications that we write for.  So we may have $450 million in medications at Geisinger Health Plan, is a rough number.  Less than one percent of the drugs that actually get prescribed are this type of drug, but they are accounting for about $150 million in cost.


So you can see why getting it right is so important.  So what we did is we took a pathway we’d been using since about 2006, which we call Proven Care, which aim to look at things as Gail said, at a high disease burden, we started them out with surgeries — (indiscernible) surgery, spine surgery, obesity surgery.  We took it and applied it to this same thing and we said, well, we can apply the same methodology, which was what’s the current literature out there?  What are the national guidelines, what’s the data that we have?  Then develop consensus around it — and really the key is then, how do you do process evaluation and transformation to actually get the right thing done at the right time?  What you realize in the last bullet is, the only way that works is if you get the patients onboard with the people giving the care.  The providers onboard with the people paying for the care.  And that’s really been, I think, the lesson learned around this.  The other thing we did as part of this, we did a collaboration with the Alliance of Community Health Plans, which includes Kaiser, includes Presbyterian, includes Intermountain.  A bunch of integrated delivery systems and sometimes not integrated delivery systems that you’ve heard about.  And we said, how can we figure out what are the right things to get rid of the 17 years and get this into real practice right away.  And what we found is, it’s really the same type of stuff — you need to build consensus, you need to have a multi-disciplinary team and teamwork is especially important.  It’s not just one person; a physician.  It’s the physician, the pharmacist, the social worker, nurses, other people.  You need to have a lot of customized education.  And this is where the patients come into play, because what I think might be the right thing to tell the patient might actually be something that doesn’t incent the patient at all to get the right thing done. With “the right thing” in quotes there.  So customizing that education so it is for everybody.


Shared data.  So one of the things we don’t do very well in healthcare is actually get — we share data, like (indiscernible) and other things, which you see six months or 12 months later.  We don’t share the data when the person is actually sitting in the room with the provider, about what’s the best thing for those patients.  And I think that’s a big part of clinical effectiveness research.  Because if you’re going to be clinically effective you need to have the data sitting right in front of you, right in time.


And the last thing I already talked about was the line incentive.  So what we found at Geisinger was what all of the other places found together.  The outcome, and the biggest outcome, what we were doing with Hepatitis C is, do you actually cure the disease?  And what we found is we can cure the disease about 99 percent of the time when we’re using the process that I just talked about — using the team to take care of the patients.  Now, the national cure rate is about 85 to 90 percent, and the cure rate of people that don’t — of our health plan patients that we can’t get into the program that we started, is only about 50 to 70 percent.  So one of the things we’ve done now is that we want all of those patients to go there as you can see.  Why is that important though?  It’s important because if you take two weeks of the drug and then you stop it, you don’t cure the disease, so you lose that.  But you have to start all over.  If you need to take eight weeks of the drug and you take seven weeks, you have to start all over if you stop.  And if you have to start all over, if you stop with $60,000 in treatment, it makes a big difference.  So it’s one of the reasons why cure rate is really important to someone like me that’s a clinician, but the cure rate is the end all.  It’s important to the patient.


The other thing we’ve learned as part of this, is you’ve learned that as you iterate over time and you set up a process like this, things that change in the literature, so that next article that comes out that says, okay, well now drug B is actually better than drug A, we can slip drug B now into this process really easily, as opposed to trying to figure out, how do you rework the whole system to get drug B in.  And the other thing we’ve learned from this through the constant iterations is, there are some people that need treatment for 12 weeks, and there are some people that need treatment for eight weeks.  When you’re talking about $30,000 for four weeks of treatment, figuring out the right patients that you only need to treat for eight weeks to get the cure, is a huge difference and one of the lightbulb moments that we had with one of our guest (indiscernible) who run this program was that for every patient they only treat for eight weeks, every two patients they do that, the next person they are treating for free.  Because you’ve saved four weeks of treatment and you basically saved $60,000, which is what it takes to treat one patient.  So it makes a big difference being able to go through and iterate this.  And really the lesson learned is if you take a total cost of care approach, if you think about it, from what is all in and you have a team, you bring much higher value and in the end the patients receive much better care, which is what the — first and foremost we want to do.  But you have the secondary gain because the cost of care is decreased.  I think in general when you’re doing the right thing, and that’s the title at the beginning:  Right Care, Right Place, Right Time.  Generally when you’re doing the right thing in healthcare, it is the least expensive thing, it’s the best thing for the patient.  Maybe times we can’t get our systems in place to get the right thing for the right patient at the right time.


And the last piece of this is, it’s completely scalable.  So we’ve — and we’ll probably talk about it in the question and answer, but we’ve been able to scale this one, across our system and then across parts of Pennsylvania for Hepatitis C.  But you can take the same methodology and use it for things like heart failure, diabetes, psoriasis, other things that are high cost with a high amount of variability.  You put something like this in play, you decrease the availability, it impacts those disease states too. So thank you.


SARAH DASH:  Thank you.  We’re going to get into opportunities for discussion and while you all are thinking of your questions to ask, and you can feel free to stand at the mic, or write it down on a card and someone will come and get it.  I want to start with something you just said, Dr. Bulger, and I think it was pretty stunning.  What you said was, if somebody who needs to take the drug for eight weeks only takes it for seven weeks, they will not be cured and they will have to start all over again.  But then there are some patients that need — literally their bodies need different courses of treatment.  So we are not talking about the blue pill and the pink pill, we are talking about the same pill with a different course of treatment.  So I wonder if you can maybe — and it had a cost implication and a patient outcomes implication.  So you’re trying to make these decisions in real time.  And I want to kind of connect the dots between what you just said and Sean’s work about real time, real world evidence.  Could you guys kind of comment on how do you kind of gain that knowledge rapidly so you can make better decisions at the point of care?


JOHN BULGER:  That’s a great question.  I think the biggest thing we learned was — and even to take a step back — healthcare is very complicated.  It’s kind of like, who knew, right?  And having a team to be able to work through this is one of the things that’s the biggest importance.  I think a lot of times you’re relying on a single condition and a single practice, even if that clinician has things like electronic health records and other things behind him.  It’s difficult to keep up with everything that’s going on.  So something like Hepatitis C, and treating it, is very nuanced because people have — each person is different — genetics play a role and other things play a role.  You’re able to tell what the right things are to treat the patient.  But having a team that includes a pharmacist, and nurses that are able to back them up and get the clinicians the right thing to do at the right time, I think has been very important.  And you’re right, I mean, the choices make a big difference and it maybe didn’t matter as much when the choices were dollars for the drug.  But when it matters even more when the choices are thousands of dollars for the drug, or when you have drugs that have a lot of side effects that using the wrong choice for the patient ends up being the wrong thing.  So those are really the two areas.  And in this case, you kind of have both of those with Hepatitis C.  But you find those same things happened with psoriasis for example.  You know, in the psoriasis case they actually — the best and most cost effective treatment many times is light therapy.  So actually having a light box, which costs hundreds of dollars. The biologic treatment costs thousands of dollars.  The issue with that is it’s many times a lot easier for the patient to use the drug.  To come to their doctor’s office, get an infusion once a month and even though that’s $10,000 and the light box is $150, it’s much easier for them.  So you have to work out being able to get the evidence, and being able to get the education to the patient so that they know what to do and understand the difference.  And the last thing just to add in that, sometimes if you don’t align the payment policy — so in the case of a light box, where we found out, when we first started doing it, the way the payment policies were set up within Geisinger, the drug even though it was $10,000 in total, was cheaper for the patient because the patients cost share because the way the benefit is set up, was less than actually going out and buying a light box, because they had to pay for the whole light box themselves.


SARAH DASH:  That goes to Gail’s point, I think, on all the research is great, but if you don’t have the systems and the incentives set up, it’s not going to happen.  John, did you have a comment on the role of real world evidence?


JOHN BULGER:   Yeah, just a quick comment.  I think the Hepetitis C example and the duration of therapy sort of underscores the fact that there is just so many questions that come up beyond sort of the fundamental comparative effectiveness questions about whether the therapy is effective.  But then there’s duration of therapy and what patients — you know, the implementation science side of it — how do you best develop these things?  And it’s impossible to answer all of those questions in randomized trials whether real world trials or traditional trials.  I think — so one thing that wasn’t around eight years ago when PCORI started and (indiscernible) this whole movement and real world evidence and big data, and artificial intelligence, and the ability to capture data from the delivery of care and become more educated about what is the right duration, what is the right kind of patients, et cetera.  And so I do think the next decade of comparative effectiveness research is going to be largely driven — or driven in a major way by advances and being able to use all of the data that just emerges from the process of delivering care, you know, that shows up in the electronic medical records, and the claims data, et cetera, et cetera.  That’s going to be an enormous source of information for these kinds of questions.  There will still always be a role for pragmatic trials, real world trials, randomized trials, but that’s more for like, age-old questions like, what’s the right dose of aspirin to use to prevent heart attacks and things like that.


SARAH DASH:  Thanks.  Gail, do you want to jump in?


GAIL WILENSKY:  I wanted to just raise a concern I have about how rarely the questions — the fundamental questions — seem to come up between patient and clinician interchangers of recognizing that there are very different ways to treat specific medical problems.  And having the conversation of these are the different ways your medical condition might be treated and to engage in joint decision making, that both lays out the pros and cons clinically, but also recognizes that there are many other issues, including the cost that the patient will bear, that are important.  I think it happens more than it used to, but I think it happens still relatively rarely.  And so getting that kind of discussion as a part of medical education and clinical training both in undergraduate medical education and in residency, is something I think that we still really need to think about, if the clinician doesn’t think about it that way.  There are some patients — like in the occasional times I confront the medical system — where anyone who I’m encountering will get to think about it also.  Most people don’t approach a medical problem that way, and so I think we’re better off than we were, but we really do have a long way to go to get people to understand, you need to step back and think about very different ways that a particular medical condition can be treated and that this ought to be regarded as joint decision making between the clinician and patient.


SARAH DASH:   So not just the evidence, but connecting the dots all the way to the end.  Eleanor, I saw you nodding, is there anything you wanted to add?


ELEANOR PERFETTO:   Yeah, I think — I agree with everything that — that my colleagues here have said.  I think the one thing that I would throw in there is consideration around the caveat around the word “best”.  You know, because we talk about, what is the best treatment for certain patients or certain patient populations and I think we still have to figure out with patient input what “best” really is.  Because we’ve been.  We have many therapies that have been on the market for a number of years.  Or if there is something like procedures or things like that, they were never actually tested to really figure out what was best in the first place.  So when we are defining “best”, we really have to have patient input into what best means and I think Dr. Alfiee said this earlier:  Best according to whom?  You know, you never asked me if I thought it was any good.  And so best is really in the eyes of the beholder and if patients never had any input in that conversation then the box is being checked that the person got the best therapy when it really has nothing to do with what they were looking for in treatment.


SARAH DASH:  And then we’ll get to the question at the mic.  Did you want to add anything, Dr. Alfiee or — you’re nodding.


ALFIEE BRELAND-NOBLE:  What my colleague just shared speaks to why we have these seemingly intractable disparities.  We come up with things, we develop them, like you said, in the lab, and then people don’t use them or don’t want them.  And so they don’t take advantage of them.  And so things don’t change.  So when I think about depression over the years, what we argue is that cognitive behavioral therapy.  I was on the committee to help develop some guidelines and that’s sort of what we settled on.  But that was based on the evidence that was available, as my colleague said here.  But what was the evidence and what was deemed as good evidence — all RCTs that had this much diversity in them.  Like, if that much.  And so we don’t have to wonder why people don’t uptake some of these treatments.  Patients, myself included, because people look at them and they say, I’m not reflected in that population, so how do I know this will be good for me?  So I think if we can build some of what has been discussed here into the process of developing and disseminating new treatments, it gives us an opportunity to even produce disparities because people will buy into it.


SARAH DASH:   Great, thank you so much.  Okay, some of you have been very patient waiting at the mics.  If you could just introduce yourself and ask your question please, that would be great.  Yes, ma’am?


AUDIENCE MEMBER:  My name is Susan [name], I’m a policy researcher at the Association of American Medical Colleges.  My question is whether the panelists have considered the relationship between the comparative effectiveness research and the personalized medicine?  So for example, the RCT — the basic assumption behind that, you assume the treatment group and the control group are in some way similar.  Also the patients within the group have to be similar, otherwise you cannot interpret the results.  However personalized medicine assumes we are all different in some way and the diagnosis and treatment have addressed the more nuances of personal characteristics,  thank you.


GAIL WILENSKY:  I think most of us have thought about it and our view is that it can complicate the decision making, but the fundamental questions that you should think about as a clinician in terms of getting to your patients the information about alternative ways to approach a clinical problem still exists.  Because in addition to whatever form of personalized medicine might be appropriate for a particular oncology patient for example, there may be other options other than having targeted — a targeted intervention of a single kind that you could use to treat that particular patient.  So yes, I think it obviously makes it a more complicated problem, but it doesn’t really change the approach to thinking about it, that clinicians and patients need to have and it’s particularly to rise above the green pill versus the bill pill.  But is there a completely different way to treat a sunlight versus and intervention, or an aggressive intervention versus watchful waiting, versus a completely different way of approaching something.  But obviously the more individualized the treatment that you’re talking about, then finding if there really are appropriate comparators may or may not exist, in which case we have different question, particularly if you’re talking about a very expensive therapeutic.


ELEANOR PERFETTO:   I would just add to that briefly that even if you’re thinking about a personalized medicine treatment, all treatments are going to have benefits and risks.  And that individual patients should have the information so that they can weigh those benefits and risks to make a choice that is good for them or good for their family member.  And so even though it might be a personalized treatment, there may be risks that the other person is just not willing to take on.


SEAN TUNIS:   A last quick point is, you know, to some degree, as much as the vision of personalized medicine, you know, the right treatment is kind of for every individual is going to be specific to that individual, we are sort of stuck for the time being that to some degree you have to extrapolate from population averages to know what’s the most likely result that you’re going to have, even though you have unique characteristics that aren’t, you know, called out.  So what I think PCORI and other folks in the field of comparative effectiveness research are focusing on this notion of heterogeneity of treatment effects where you can identify subgroups of patients within studies that have specific characteristics and the risks and benefits might be different than for the overall population.  But I think it’s important to sort of not fall entirely victim to the notion that the only way of picking the right treatment for myself is to know what patients exactly like me have experienced in the past.  Because that’s not the way evidence is aggregated and extrapolated for making treatment decisions.  You know, we can move in that direction.


JOHN BULGAR:  Yes, I would just add to that, because I think it plays with what we said about what’s best and what’s right.  I think you can add into that, we talk a lot about value and you know, all those questions are — the answer should be to the patient.  So what’s best for the patient, what’s the value to the patient, what’s right for the patient.  Personalized medicine — all of these questions medicine, none of it is black and white, its all shades of gray.  I think personalized medicine is going to maybe make those shades get closer, but it’s still going to be shades of gray.  It will just be another tool which you can add in.  It’s why I think, you know, working on the delivery science is so important because then if you’ve worked in delivery science, you can add those tools in and you can increase the wealth in the shared decision-making discussion you have between a physician and a patient as opposed to having — right now, I think personalized medicine I think to many conditions is just kind of out there.  Not really sure how to use it.  And it does have this black and white feel to it when it really isn’t.  It’s really just another tool to be added in to all of the other tools we have to take care of patients.


SARAH DASH:  Great, thank you all.  Okay, so I just want to point out, we have — it’s great to see so many questions.  We do have folks at the mic.  Introduce yourself, keep your question brief and we’ll try to get to everybody as much as possible.  Yes, sir, you’ve been waiting.  Go ahead.


AUDIENCE MEMBER: Bob Griss with the Institute of Social Medicine and Community Health.  I’m wondering how seriously we’re actually taking — our healthcare system is looking for alternative treatments.  It’s one thing to say that comparative effectiveness research helps choose between the red pill and the blue pill, which I guess corresponds to our political spectrum.  But are we actually looking for preventive strategies?  Are we actually looking for non-clinical strategies?  Are providers allowed to prescribe non-clinical interventions?  I think there are a number of questions around comparative effectiveness research that has to focus differently on alternative treatments.  Especially for things that we don’t have cures for, like cancer.


SARAH DASH:   Great, thank you for your question, I think you made your point.  So can comparative effectiveness and patient-centered outcomes research focus on different kinds of non-medical or other kinds of treatments?


GAIL WILENSKY:  The answer is yes, whether or not somebody is going to be willing to pay for it will depend on the system in which they receive care.  So if you’re at a Geisinger that covers all of someone’s healthcare for a preset amount of money, and there is evidence that there are alterative medicine approaches or other approaches, they may well be willing to consider it.  If you are a traditional fee-for-service, it depends on whether or not there is evidence and something about the payment system that you’re in.  So this is information that’s useful to have, how it will be used will depend very much on the kind of system or lack thereof that somebody is involved in, because the financial incentives will be very different.


SARAH DASH:  Mike, did you have a question?


AUDIENCE MEMBER:  Thanks, Sarah.  Mike Miller, I’m a physician and health policy consultant and I spend about half my time in an organization called Health Women now.  So my question is about sex differences in CER.  How that’s approached, because now FDA NIH have requirements for including women in clinical trials and research.  And particularly if you can comment about how that information around sex differences can influence or should be considered for benefit design or coverage roles and how CER can influence that.  Since we know there are certain diseases where there is different symptoms for women, versus men, or different therapy effectiveness.


SARAH DASH:   Thanks Mike.  Anyone want to take that? It’s a big question, but diversity in research and —


ELEANOR PERFETTO:  Mike, it’s been settled, men and women are different.  No, it’s a great question, but I think it’s not a different question than other aspects of diversity in the population.  We’ve done a poor job of capturing all the sub populations that are out there and part of comparative effectiveness research and patient-centered outcomes research is to try to do a better job of that and I think Sean pointed out that we can’t do a clinical trial that captures in large enough populations every single characteristic that we might be looking for, so we turn now to big data, real world evidence, AI, to see if it can help us with some of those kinds of things. But yes, by all means, we have got to do that kind of work to really have a better understanding of all of those kinds of diversity issues.


GAIL WILENSKY:  It’s really helpful if you just have people recognizing this and appropriate question to ask — there has long been a dearth of women in many of the clinical trials recognizing either there is or isn’t evidence about whether there are any gender difference, but knowing that’s a relevant question and at least being alert and sensitive to potential outcome differences is important. So a lot of times with comparative effectiveness information, it could help clinicians and patients know the questions they should be asking, even if the information may not be readily available.


ALFIEE BRELAND-NOBLE:  I also think there’s one other thing that maybe we’re not saying, which is — my colleagues have alluded to it, and I think that is, we have to fund researchers who are interested in asking these questions across diverse populations, right?  so for the people who have been ostracized from the field in terms of being clinical investigators, we have to reward the people who come to the table — I’m saying this as a reviewer, with the questions already on their mind.  That they are going to do these analysis.  And so if we can build that — that’s a systemic issue, which is one of the reasons I value PCORI so much is because they allow people like me to shove those questions in there.  Not a secondary or tertiary question, but as a primary question.  The more we can do that, the more it becomes natural, the less room there is to question whether we have enough of the right people at the table to answer the questions that we want to ask for the widest range of people.


JOHN BULGER:  And I’ll just add, I mean, I think one of the things PCORI has done, it’s gotten researchers together to start to normalize the data from different electronic health records and different data sets.  So that those data sets now exist, that male and female exist in the global data set and where one data set would have had male and female as one and zero, another one was F and M, and other — those have all come together because of PCORI and I think people who want to answer questions in the future and ask those questions, it’s now much easier for them to go and use big data or however they want to do it, to answer those questions.


SARAH DASH:  I think we have two more questions at the mic.  So we’re getting to the lightening round.  So state your name and your question, please.


AUDIENCE MEMBER:   Phil [name], I’m a PCORI ambassador and I loved your history about death panels.  I would like to bring up the fact that we’re getting another death panel called quality adjusted life years.  And one of the problems with comparative effective research has been pointed out with women — seniors with complex medical conditions are often not included in those.  And the question is:  How do you use big data, how do you use groups that aren’t included in the normal comparative effective research to save people from death panels and quality adjusted life years and step therapy?


SARAH DASH:   So let me — we also had some questions on the cards, and Sean, you kind of alluded to this.  So I think there’s a lot in that question, but I think part of the job of the Alliance is to welcome the elephant into the room.  We started off and Sean you talked about a name change, because you sort of said, well, there was this problem with “death panel” so we changed the name to PCORI.  But the question I wanted to ask is:  Is this actually more of a paradigm shift from like this idea that if one treatment supposedly works better than another one, someone is going to get denied care, as opposed to like, let’s figure out how to integrate the patient voice into the entire fabric from start to finish, which is kind of what I’m hearing on the panel.  So I want to take your question, but I want to broaden it.  Is this a total paradigm shift?  Eleanor, I think at the beginning you alluded to we’ve seen a lot of changes over the last ten years of really seeing the need for more of the patient voice.  So do the best you can with those questions.


SEAN TUNIS:  Yeah, this is a dangerous question because I think almost every answer is going to be wrong.  I mean, I would take the point, without having an answer to it, that the quali is extremely problematic, but it’s highly functional for decision making.  And you see that.  So I’m not saying that to defend it, because I think it’s hugely flawed in that it sort of assumes certain homogeneity and parallels of what matters to which patient groups.  And I do think that for elderly, disabled, et cetera, et cetera, it — to the extent it actually informs decision making about what’s paid for and how much things are paid for.  It’s really got some kind of flimsy underpinnings.  And there’s lots of folks trying to work now, I think, on alternatives to the quality ways of doing things.  As far as the paradigm shift, you know, it’s — my personal opinion is that something was lost when the name was changed from CERI to the Patient-Centered Outcomes Research Institute, and something important was gained.  So I just think it’s time to sort of re-look — that’s why I wanted to sort of do a backward look of where do we start this whole thing in terms of these — some of these enormous gaps in evidence that make it difficult for anyone to make a decision — whether it’s a payer, or a clinician, or a patient and address some of those flaws.  And I think we’ve sort of been partly successful, but at the same time, the heavy emphasis on patient-centeredness I think left some of the limitations in the body of evidence poorly addressed.


GAIL WILENSKY:  I want to echo and extend that thinking.  The notion that what was important to patients was never excluded from any kind of thoughtful comparative clinical effectiveness.  It is to recognize that treatments can have different outcomes and those different outcomes can be both medical, clinical, functional, economic and it is complex decision as to how do you go about making a decision that ought to be between the patient and the clinician?  I think the notion that what we know about what works when, also influence a payment and reimbursement decision is naïve at best.  That is a reasonable factor to come into a decision.  But some of the payment systems make it much easier to have tradeoffs like pre-paid systems, as opposed to a fee-for-service, where it may or may not be covered, but there are other incentives also going on in terms of what might look most attractive from the clinician’s point of view and what might look most attractive from the patient’s point of view.


SARAH DASH:   I think just to kind of point back to John’s example of the light box versus an expensive drug.  That if someone’s got to pay out-of-pocket for a light box that actually worked better versus a drug that might — that’s where some of the payment and coverage decisions can also factor in.  Did you have something to add, Eleanor?


ELEANOR PERFETTO:   Yeah, I think the quality is — it’s just a controversial point and I think we kind of have to get past it and say, we all need to come together and figure out what is the most reasonable way that we’re going to all talk about how we make these choices.  And then if you have a mechanism that is extremely hot-button, that’s extremely controversial, and we know has significant flaws, why would we be whetted to it?  And people go around talking about how it’s the standard — I’ve looked through the literature for the last few months to figure out where it came from that the quality is the standard, and somebody just decided to call it that.  And so without evidence that we really have something that works for us to help us prove that we’re making the best decisions, then we have to find out what that really is, because we don’t have that evidence.  We know that there is a lot of significant flaws, we know that there is a lot of significant controversy.  Let’s all get together and figure out a better way.


SARAH DASH:  We are running close to our time.  We do have time for one more question.  For those of you who did not get a packet, and we apologize, the materials are already online at allhealthpolicy.org, so please check that out and don’t forget to fill out a blue evaluation form before you head out.  We will take your final question, ma’am.


AUDIENCE MEMBER:  No pressure on me at all.  My name is [name] and I was a clinical research epidemiologist at Oregon Health and Science University and I managed two career development NIH grants there as well.  I do a little health consulting here, but I wanted to point out, a lot of what Dr. Alfiee and others have said have resonated with me are from those human-centered design.  And in every other industry, that’s where we start.  And I realize that in five years of hearing scholars present their research ideas and compared to the effectiveness in others, as we have a large center there, no one talked about including the patient in research design.  Never once in five years was a single patient there.  I’m wondering for those of you who serve as reviewers, is that going to happen?  Are we going to start valuing the voice of patients in designing that clinical research so that it’s more accurate and reflective of the population we serve?


ALFIEE BRELAND-NOBLE:   I think it’s critical, but I think also those of us who do that are the lone wolves at the table.  Because our colleagues — and I’m not disparaging anybody — but that’s not what’s important always to our colleagues.  Is how were the patients involved with the design?  What they want to know is, is it rigorous?  As if somehow there is this dichotomy.  It’s rigorous, or patients were at the table helping with the design.  And that’s really not how it is.  And so for people like me and other disparities researchers in particular who care about social determinants of health, those folks I think are often at the forefront of setting at the table with patients, asking the questions:  What kind of measures are meaningful for you?  We want to explain the design to you.  That’s part of why we came up with the curriculum that we came up with to build capacity so that patients are empowered to be at the table.  They are not just sort of sitting there like window dressing.  Which again, I think is why my colleagues at PCORI asked me to come participate in some of these events, unless I get too rowdy.  So I think it’s a good question.  I think that there are enough of us who are trying to be that voice.  I think with PCORI we’ve been empowered to be that voice more at other tables.  I do review for lots of folks, I’ve stopped reviewing for one, I won’t say — they will remain nameless.  But I do think that there are those of us who are trying to center that voice, particularly in terms of research in review.  And evaluating people on that metric.


SEAN TUNIS:   One quick thing to add is, I think that there are rapidly growing islands of great sort of — are at an advanced stage of not just engaging a meaningful — engaging patients, patient advocates in research design, prioritization, et cetera, but more and more we’re seeing patient advocacy organizations that are leading the research and inviting the researchers to support them so the patient — parent project on muscular dystrophy, the National Hemophilia Foundation — I’m sure Eleanor, you —


ELEANOR PERFETTO:  And I was about to say the same thing.  The barn door is open, you know?  Patients now are much more empowered and it’s their expectation that if you’re not doing the research with them, you’re doing it wrong.  And they are going to be very vocal about telling you that.  They are going to tell you that.  They are going to be out there.  And they are funding their own research, they are part of the conversation, they are sitting at the table.  And that’s going to grow because they are certainly not going to just walk away and say, “Oh well, they wouldn’t let us do it.”  I also think the other thing too and I think Dr. Alfiee said this also, there isn’t an either/or.  It’s not patient-centered or rigorous.  You can do really good patient-centered research that is extremely rigorous.  Do we have an evolving science on how to do that really well?  Yes.  But does it mean that it can’t be done and that it’s either/or?  Absolutely not.


SARAH DASH:  Thank you. On that note, I think we need to do a whole other follow onto this on redesigning research and that there is not an either/or between is a rigorous or is it real, right?   To patients.  So with that, unfortunately we are out of time.  But I would like to thank all of our panelists for joining us today, thank all of you for joining us today.




And look for the next briefing in the series.  Thanks.