A.J. Holmgren, PhD, an assistant professor in the department of medicine at UC San Francisco, discusses EHR quality measurement, quality of care and patient outcomes. This episode is part of the Health IT series by the AMA-MSS Committee on Health Information Technology, hosted by Kristofer Jackson, medical student at the University of Toledo College of Medicine.
In this episode of Making the Rounds, A.J. Holmgren, PhD, an assistant professor in the department of medicine at UC San Francisco, discusses EHR quality measurement, quality of care and patient outcomes. This episode is part of the Health IT series by the MSS Committee on Health Information Technology, hosted by Kristofer Jackson, medical student at the University of Toledo College of Medicine.
Listen on the go to the full episode on Apple Podcasts, Spotify or anywhere podcasts are available.
Jackson: Hello and welcome to Making the Rounds, a podcast by the American Medical Association. Today's episode is part of our health IT series from the AMA Medical Student Section Committee on Health Information Technology. My name is Kristopher Jackson, I'm a medical student at the University of Toledo College of Medicine and I'll be your host for today. Today we're joined by A.J. Holmgren, an assistant professor in the department of medicine at UC San Francisco. Hello and welcome, Dr. Holmgren.
Holmgren: Hi. Thank you so much for having me.
Jackson: Thank you for coming on. So just to get us started, can you tell everybody who's listening at home how you're involved in health IT and what you're currently working on?
Holmgren: Absolutely. So, I am an assistant professor of medicine at UCSF and a health services and informatics researcher. My primary appointment here is in the Center for Clinical Informatics and Improvement Research, as well as a faculty affiliate at the Bakar Computational Health Sciences Institute. So, research on health IT, evaluations of health IT and thinking about how clinicians use health IT is my day-to-day bread and butter. It's what I do, I'd say, 80% of my waking time in some capacity.
My research has a couple of distinct streams within the health IT space. The first, and what really motivated me and got me into this space, is interoperability. So, the electronic exchange of data across health care organizations and settings for the delivery of patient care. This has been both the technical challenge as well as a policy and market failure challenge for years now but it's a service that we should absolutely be providing to patients to help deliver the best care. The analogy I always like to use is that my ATM card will work in outer Mongolia but I cannot get my records sent from UCSF to Stanford.
The second stream of my research is applying observational quasi-experimental methods to health IT research. So, I'm a PhD researcher. I come from a very interdisciplinary health policy background and a lot of my training is in what we think of as applied micro-econometrics, and this is a big focus there. So, I spend a lot of my time trying to find places and sources of randomness in the world to answer important questions that we can't use randomized control trials on. So, we can't randomize EHRs to hospitals but we still really want to know what aspects of EHRs improve care.
Finally, a growing part of my portfolio is thinking about how to use EHR audit log metadata, which is this massive source of sort of secondary data that is generated every time a clinician interacts with an electronic health record. It tracks every single click, mouse movement, everything you do in the EHR, to measure clinician work, examine physician decision making and especially focus on clinician productivity and well-being. How much time they are spending documenting in the EHRs. This has obviously been a big area of focus lately in the policy sphere given the sort of ongoing discussions about clinician burnout generated both by EHRs and the pandemic. And so, it's sort of becoming a bigger and bigger part of my research portfolio.
Jackson: That seems very interesting. It's actually kind of in the ... Dr. Anupam Jena, who we interviewed earlier, I think, last year does stuff like that as well but his is far more out there. Like the reaching of car accidents after Fast and Furious movies and stuff like that, but just looking for randomness—
Holmgren: Bapu is the absolute king of that kind of quasi-experimental finding, that source of exogenous variation. He's an inspiration for sure.
Jackson: Yes. So, what would you say were the challenges that you experience in your day-to-day work?
Holmgren: I think the challenges of this type of work are many. In terms of the causal inference, the absence of randomized control trials, IT is almost always a way thought think of as endogenous. So the decision as a health care delivery organization to invest in some sort of complex, expensive, difficult IT project because you think it's going to have big benefits to your patients or your employee physicians or your nurses is almost always correlated with some other measure of quality. So obviously if you are the Mayo Clinic, you have both a lot more resources and perhaps a lot more drive to be the best health care delivery organization in the world compared to maybe a for-profit hospital has a little bit less of a sort of motive because they are motivated by other forces.
The resources aspect is really important too. There are a lot of safety net clinics out there that don't have anywhere near the level of advanced IT that big academic medical centers have. And that's not because they are lower quality or they don't want to. It's just they have less money; they have less access to talented IT staff. Which when we talk about anything related to health IT, it's not just buying software, you need a whole host of people to sort of maintain that software, to run updates to make sure everything's working and those are expensive and they are hard to find in rural areas.
And so, trying to find some clean identification of a causal effect, when you want to ask a question of, what does, say, clinical decision support do to improve quality is really, really tough. Because if you look at just the naive correlation there, you're going to find, oh, well, all of these hospitals or care delivery organizations, or multi-specialty clinics that have adopted advanced IT also have better patient outcomes. But is that because of the IT or is that because they have a lot more money and resources to throw at any given problem? And trying to tease out those effects is really, really challenging. And I think that's both a challenge to our knowledge base but it's also what keeps me interested in this work. So, it's really a great sort of creative challenge as well.
Jackson: One of the challenges that I know that you've talked about in your papers is that there's not a lot in the way of evaluations for EHR performances. There are certain EHRs that I know that we as health care professionals have opinions about but on like a clear level there's no this one is better than this one here. So how do you go about evaluating that if you are the Mayo Clinic and you're getting ready to make a $10 million investment or whatever it is into some health IT project?
Holmgren: I do think some EHRs are better than others on various aspects of them. The big market leading developers, Epic and Cerner on the hospital side, there are a lot of reasons the market has coalesced around those two but quality of some aspects of the software is clearly one of them. That said, I don't think any one vendor is a clear best of class enterprise IT software solution for all domains obviously.
When we think about sort of evaluations of EHRs, I think it's useful to sort of think about it in a quality framework. So, the Donabedian quality framework, which is very classic health care model of care quality, likes to break things down into sort of three big buckets: structure, process and outcome quality. Structure quality is what we've been measuring a lot with EHRs. Just sort of, do you have an EHR? Do you have the sort of physical space, the physical environment, the tools to deliver the best quality care? Process outcomes are for when we know what the evidence-based thing to do is. Is it happening? Which is to say, when we know an order is being entered that would generate some sort of adverse event for a patient, is your EHR clinical decision support alerting the ordering physician, hey, it's likely to generate an adverse event for this patient. Then outcome quality is what we think of as traditional health care quality. This is mortality patient experience. You know, did the patient recover?
And so, I think what we need to transition from are these measures of structure quality, i.e., do you have an EHR, what functions does it have, what can it do to more measures of process quality. And I think we're starting to see this in the quality measurement space, both by sort of the various reporting programs required by the federal government. So, the Office of the National Coordinator for Health IT, ONC, has the EHR reporting program. Various other firms have sort of ongoing process quality measurement. This includes the Leapfrog Group and their clinical decision support medication safety quality. NCQA, for those participating in the patient-centered medical home program, has several process quality measures. So, we're starting to see a movement towards this but I think just getting those quality measures in the first place towards more process quality measures is where we need to be.
Outcome quality, really, really difficult to attribute to any particular aspect of IT. So, I think as much as that's what we truly care about, is the patient experience, the patient outcomes, I think moving from structure to process quality measures is going to do a whole lot to improve our understanding of IT capabilities and how they sort of influence physician decision making towards delivering the best quality care.
Jackson: Okay. But you did note, I think it was the association of hospital public quality reporting with electronic health records, that after hospitals got a negative grade there was a significant uptick in their quality and their safety reporting. How are they doing that if this is so hard to keep track of and the patient outcomes are hard to determine here?
Holmgren: I mean, I think this gets down to a sort of fundamental fact about policy making in health care is that it's really, really difficult. Lots of different stakeholders have very different priorities. So, one of the things that a lot of different groups have tried to do is use the levers that they have to incentivize improvement that they care about.
The Leapfrog Group is a really interesting one because it essentially started out as an employer based group, attempting to measure the quality of care. Employers obviously are paying for most of this care. Most people who get significant health care in this country are getting it from a private insurer and most large employers are self-insured, so they are paying the claims themselves. So, they have a pretty strong incentive to ensure that their employees are getting the highest quality care because they are paying for it. And what the Leapfrog Group does is it sort of publishes these transparent hospital rankings and scores. You get like an A, B and C based on your various safety features. One subset of that is their evaluation of EHR safety in the form of clinical decision support to sort of reduce adverse drug events from inpatient drugs. And so, what we did in this study was to look at, okay, so hospitals that are just barely missing the A grade, do they seem to respond and the next year do they improve more than hospitals who seriously missed the A grade or who got an A or above? And they do. And so, what we see here is that hospitals really actually do care about this sort of thing.
Now, I don't think that this is because that they're worried about running out of money. I don't think there is any serious concern. Hospitals have significant market power. I don't think patients find these sort of grades extremely salient. You kind of have to click around to get to the hospital EHR safety grade on the Leapfrog website, so it's really unlikely that as a patient you're going to say, "I'm actually going to go someplace else because this hospital got B grade."
What I think hospitals really do care about is being perceived as the best highest quality provider. If you walk around any major city, you'll see all sorts of ads ... they want to be perceived as the best quality of care. They also want to use it in advertising, sort of be perceived by their peer group. They want to attract talent. They want the best physicians; the best physicians want to work at the best hospitals.
And then similarly on the physician side, most clinicians really want to deliver the best quality of care. We have this incredible resource in health care of the professionalism and intrinsic motivation of physicians and health care delivery organizations to deliver the best quality of care. And I think that's a huge resource that's a little bit underutilized. Turns out when we publish these sorts of public quality reporting, then you can use that sort of intrinsic desire to be the best to sort of motivate these organizations to improve along these lines of care quality, which I think is a really powerful result.
Jackson: Okay. And just to follow up on that, are these hospitals improving upon their quality of care themselves or are they just kind of leaning on their vendors and their EHR rep saying, "Hey, we're not doing good enough so we need this to be improved or that to be improved"? What do you think is actually happening here?
Holmgren: So, it's probably a little bit of both. A significant amount of EHR customization happens at the site level. So in that study where we looked at the Leapfrog Group's data, we found a big amount of variation even within the same EHR vendor. What that tells us is that hospitals do make a significant amount of customizations, especially in regards to clinical decision support. Deciding what alerts to keep on, what alerts to turn off. Trying to balance the sort of tightrope act between clinicians who feel burdened by alerts and getting too many alerts leading to alert fatigue and then just dismissing them all, versus the actual safety concerns of trying to prevent adverse drug events.
So in this I imagine it's probably sort of a co-process between the vendor, the hospital's IT staff and the various consultants they likely have to say, “Okay, we did poorly on this, we got what is essentially amounts to the B grade and we want to get the A grade next year. And so how can we do this?” And that partially involves turning on more alerts. And it partially involves sort of just an ongoing process that says, where are we at with our decision and our sort of a framework for what alerts we send and what alerts we turn off. And I think it sort of nudges them to air a little bit more on the side of safety.
Jackson: And are there any other carrots or sticks that we can use to kind of help hospitals improve? Is there anything else that we can hold over their head that is really going to motivate them, in your opinion?
Holmgren: Yeah, I think so. And I think there are some programs that are sort of trending down on that path right now. On the interoperability side, the 21st Century Cures Act rules, which are recently implemented by the Office of the National Coordinator for Health IT, are starting to go down the lines of making it essentially illegal to information block, to purposefully create barriers to sharing patient information when it should be shared.
And this has been sort of a big problem because interoperability is a technically challenging thing but that's not the main reason it's not being done. The main reason it's not being done is because hospitals and EHR vendors don't see a business case to do so and that hurts patients in the end. So starting to build in things like the sticks there, as in fines, as in significant penalties, potentially CMS reimbursement reductions later on down the road as a potential way to motivate hospitals to start sharing that data electronically and start doing what we want them to do as a way to correct that market failure and say, "Hey, I know you don't want to share data with competitors but you're going to have to."
Jackson: And along those same lines, one of the more astonishing facts I've seen in some of your papers is that it takes the average resident about 46 minutes per patient with a lot of that time coming after hours on their own time. What's happening with that? Is that the residents? I know some of it's the residents and some of it's the system but what exactly is the main cause of this and how can we reduce this so that our health care workers aren't spending an hour each patient on EHR?
Holmgren: Yeah, absolutely. I think it's a really important question, especially given how much residents are already dealing with. And this extends to attending physicians as well. We see really, really high EHR times compared to sort of other countries. So, I think there's a few things. The first is that most EHR interfaces are not paragons of user experience design. If you've used one, you kind of know that it looks maybe more like Windows 95 than sort of a best of breed iPhone app. The second is that it's important to remember that a lot of the work that's being done in the EHR is not mandated by the EHR. Nothing about the design of an electronic health record says you need to document this much. That stuff is mandated by policy, regulations, billing, quality assurance, research. All sorts of different things that are really not at all technical requirements. Those are designed by people.
A recent study I did in JAMA Internal Medicine highlights this, special ambulatory clinicians in the U.S., so physicians and advanced practice providers, spent nearly twice as much time per day in the EHR as their counterparts and peer countries, primarily in Canada, Western Europe and Australia. That is a huge amount of difference in terms of the EHR work they are doing. And if you talk to physicians in Canada, for example, and ask, "Well, what do you document in the EHR?" They'll say, "What the next clinician needs to know to treat the patient."
And if you ask physicians in the United States what they document in the EHR ... The U.S. model, a lot of our documentation's required for billing. So, there is a multi-pay environment in the United States. We have a lot of different billing companies. We have a lot of different payers. And each different payer has different preferences about what is documented. They have different sort of setups for what is going to be denied and so physicians sort of have to document to the maximum payer because for the most part they are not sure who is going to be billing for this service in a lot of cases and so they want to make sure the claim doesn't get denied.
There's also a ton of documentation that goes in that's probably mostly defense against the possibility of litigation in malpractice claims. So, you want to document everything you did in case something comes back on this. We have a particularly litigious environment here in the U.S. with respect to malpractice. And then there's all this documentation to meet your various quality metrics for your private payer as they likely have some sort of quality program that you're enrolled in. CMS has the MIPS program, which requires significant documentation. If you're in an ACO you have that, if you participate in any of these various quality measurement programs. So, I think we need to realize that EHR work is a sociotechnical process. It's not all on the IT, it's about what are we asking the IT to do and what are we asking the physicians to do through that IT. The drivers of work demands are more often policy choices, implicitly or explicitly, than the demands of the tech.
There are some movements to address that. The changes to the E&M coding and billing requirements in 2021 are sort of an early move to try to reduce the amount of documentation needed for billing. Early research has shown that hasn't had the immediate impact we kind of hoped. Turns out documentation patterns are very sticky. They are informed by medical culture and training, and the way that you're trained to document is often the way you document for the rest of your career. So, it's likely going to be a long process to try to move that. It's going to take a concerted effort through a whole lot of different stakeholders, probably led by CMS but including private payers, hospital organizations, professional societies, EHR vendors.
But in my opinion, there's only so much mileage we can get out of making EHRs better or providing better documentation tools, or scribes or AI digital assistance. We need to at least reckon with the root cause and ask, what are we getting out of this documentation? Is it worth it for the cost to physician time and well-being? It might be. Research, quality, et cetera, those things have real benefits but there's a tradeoff there, and we need to quantify the amount of time and effort we're splitting in there to the benefits we're getting out of it.
Jackson: Just a general question that I'm sure you're tired of hearing, but how has COVID impacted your work and feel free to break that down?
Holmgren: Yeah, absolutely. I mean, I think there's a couple of things that COVID has really done. Professionally, the first and most obvious is the expansion of telehealth, which is in the health IT space obviously really big. There's all this care being delivered virtually now at absolute explosion. Still at a much higher level today than it was in 2019.
So, we're currently working on a study to compare EHR use in virtual versus in-person visits here at UCSF using very, very granular EHR data to examine both changes in care patterns. Like, are physicians more likely to order more diagnostic testing when they are delivering care virtually compared to being able to see the patient in person? What is the lack of a physical exam due to your decision making when you're in that visit, as well as in terms of comparing the EHR work to in-person care? So early on the pandemic what we heard a lot was, I've shifted so much of my work to after hours because I can't document while I'm doing a Zoom visit. These are the sort of things that we're interested in is, how does telehealth and virtual care sort of change EHR work.
A second topic that I'm really interested in in the post-COVID world is asynchronous communication, so secure messaging between patients and clinicians and the EHR patient portal. So MyChart in the Epic parlance. In the COVID public health emergency, these have become a billable service in more circumstances and their use has absolutely exploded. You can see this all over the place where physicians are saying, "I get so many more patient messages now." The downside of that is this is all new work for physicians. Now suddenly there's this new demand on their time from patients. The upside is it could be a really great way for patients to access care in a low-cost convenient setting.
Say, you cut yourself and you want to know, hey, does this need stitches? And are they ED stitches or can these be urgent care stitches? And so, you can just take a quick photo, message it to your primary care office or a local urgent care and someone gets back to you on what to do. And maybe that averts an ED visit, which is really miserable for most patients because you're probably going in and waiting for several hours, and it's really expensive on the health care sector side. And that would be really great.
But right now, we know very, very little about these messages other than they are absolutely crushing several physicians who are receiving so many of them. We don't really know; are they compliments or substitutes for in-person care or asynchronous care? Can they replace a visit or are they just adding more work? And if they are compliments, we're not really sure how much health we are getting out of them if we're going to pay for them.
And this is a big area of study in the future for me because that sort of policy is being made right now. We're making decisions right now. After the COVID public health emergency order is finished, what's going to happen these messages? Are we going to keep paying for these? Are we going to ask provider organizations to sort of reorganize themselves to add this new stream of work? Because if we don't, I fear that, as much as physicians really, really want to be able to provide this service, if it's not getting reimbursed and at the end of the day you've got a limited amount of time, things that are going to get dropped are the things you don't get paid for. And so, it's just going to become, well, come on into the office or it takes so long to respond to them if we are not able to carve out time to do this work. It won't be a useful service anymore. And so that's one way I am really interested in sort of try and quantify what are the health benefits from these messages.
Jackson: Yeah, I could see how that could arise as an issue where ... It could be very helpful, like, "Hey, I need my prescriptions refilled," but I've seen a lot of messages with physicians who I've been working with where it's like, "No, you have to come in for that. I can't do this through a message. We need to actually see you to be able to diagnose and treat this," and some patients, especially in the height of the pandemic, were not happy about having to come in.
Holmgren: And I think that's another question is and you can really see how policy shapes technology use. We have this really powerful medium to access your physician asynchronously if you have a non-emergency thing. I would much rather just send an email and say, "Hey, do I need to get this checked out?" But if policy doesn't allow it to happen, whether it's not allowed regulatory reasons, whether it can't be billed for it, then the technology doesn't do anything for us if we have some sort of sociotechnical process that keeps us from using it.
Jackson: Okay. When I'm an attending physician in 10 to 15 years, how do you think health IT and health care are going to be changed?
Holmgren: I mean, I'm really excited about health IT in 10 to 15 years. I think it's best to think of EHRs as what a lot of IT researchers call a general purpose technology. And you see this all the time across a ton of fields is that some new, extremely hyped-up technology will come in and it'll massively disrupt the workflow in an industry and then we'll say, "Oh, but we don't see any more productivity or productivity actually went down." And this happened with personal computers coming into offices. It happened with the internet. It happened with a ton of things throughout history.
And what these are … they are general purpose technologies and what they require is sort of a process of innovation and co-invention in workflows, which is to say, right now what we basically did was digitize the paper workflows in medicine and now you document on a computer instead of a piece of paper. That is likely not the most effective use of this new technology. In fact, it's probably counterproductive. And as we've seen, we're slowly starting to build more and more actual useful technologies than the EHR, it's going to take a little while for medicine to figure out, okay, what's actually the best way to document? What's actually the best way to get value out of having digitized all this data? And so, sort of having that process of co-invention, of innovation, of allowing sort of physicians and other users of EHR to figure out what is working and what is not and to sort of build a new regulatory and sociotechnical system around that makes me really excited about this and I do think we're going to end up with a really great system eventually. And the next thing that I think is really exciting is that, thanks to new rules in the 21st Century Cures Act, all EHR vendors now have to build in standardized APIs to pull data out. This allows a third-party developer to create a more standardized application that builds on all this digitized clinical data in the EHR to do something cool. Maybe that's predictive analytics, or machine learning, or some sort of AI application to help clinicians make decisions. Maybe that's something for patients to allow them to extract their data out of the EHR and incorporate it into their own personal lives. Regardless, it's finally starting to get to the point where EHRs are a platform that digitizes data and that we build cool things out of rather than being the end all be all of health IT in the clinician world.
So that's why I think there's a lot of focus right now on documentation burden and the negative impacts of EHRs but I think in 10 to 15 years hopefully we'll be transitioning into a period where we've sort of accomplished that co-innovation and co-invention, figured out how to sort of work with EHRs and we're starting to get really, really cool ways for the computers to supplement the physician decision making.
Jackson: Okay. And this will be our final question. Do you have any channels where people can connect with you and follow your work if they are interested in what you're doing?
Holmgren: Absolutely. I'm on Twitter a lot, probably too much. But I tweet a lot about my research, health policy and technology and informatics in general, and some economics, and a lot of photos of my dog. My Twitter handle is @AJHolmgren, so first and last name. And so, feel free to connect and happy to share with anyone who is interested in my research.
Jackson: Or just interested in the dog.
Holmgren: That too.
Jackson: Well, everyone, that's all for today. Thank you for listening and thank you for your time today, Dr. Holmgren. This has been Making the Rounds, a podcast by the American Medical Association. You can subscribe to Making the Rounds and other great AMA podcasts wherever you listen to yours or visit ama-assn.org/podcasts. Thank you for listening and goodbye.
Disclaimer: The viewpoints expressed in this podcast are those of the participants and/or do not necessarily reflect the views and policies of the AMA.