Anupam B. Jena, MD, PhD, the Ruth L. Newhouse Associate Professor of Health Care Policy at Harvard Medical School and a faculty research fellow at the National Bureau of Economic Research, discusses using creative thinking to devise natural experiments that can help guide health care and spending.
Randomized clinical trials are widely considered the gold standard for determining the best options for medical care. Yet RCTs are rarely aimed at examining broad health care policies or answering urgent questions. When the scales of time, money, and study participants come up short, wide-angle curiosity and big data can be harnessed to devise natural experiments that deliver surprising insights into medicine and science. In fact, thinking unconventionally while applying firm frameworks of statistics, sociology, and economics can highlight what does — and doesn’t — work in health care and help guide spending.
Anupam B. Jena, MD, PhD, is the Ruth L. Newhouse Associate Professor of Health Care Policy at Harvard Medical School, a physician at Massachusetts General Hospital, and a faculty research fellow at the National Bureau of Economic Research. His responses to our questions below are excerpted from the Executive Education at HMS webinar, “Natural Experiments in Health Care: What Really Works.”
Edited and condensed for clarity.
My background is in economics and medicine, so I naturally gravitate to questions that are at the intersection of both of those fields. I tend to think about how people make decisions, whether they are doctors, patients, or companies, and then try to use large data to gather some insights into questions that people may not always think about.
A few years ago, my wife ran a five-mile race that started here in Boston near the Seaport. The race route happened to go near Massachusetts General Hospital, where we both work, and I had planned to park at the hospital to see her cross the finish line, but the roads nearby were blocked off, so I went back home. When I told my wife I was sorry that I couldn’t see her, she asked, "But what happened to all the people who needed to get to Mass General that day?"
So, just a few weeks later, my colleagues and I started to look at whether or not there's any evidence that people have delays in care on the days that big cities hold marathons. And what we found was quite striking. If you look at people who have cardiac arrest, which is when your heart stops, or a heart attack on the day of a major marathon in the US, their mortality rates are about 10% to 15% higher than on any other day of the week in the surrounding weeks.
Those effects are concentrated among people who live along the race route. You don't see the effects on people who are in the same area, but just not on the race route. And the other thing that we showed was that ambulance transport times go up about 20% on the mornings of the marathons when the roads are closed, but they return back to normal in the afternoons when the roads reopen.
A randomized clinical trial just tells us, on average, this treatment works better than another treatment. But it doesn't give you any guidance as to which patients might actually most benefit from that treatment and which patients might have no benefit, or whether some patients might even be harmed. It just tells you that if you had to throw a dart at a board, you'd be better off picking the surgery or the procedure that's shown effectiveness in the clinical trial.
What happens if you can't conduct a randomized controlled trial, either because the number of patients are too few or because it takes time and money to collect this kind of data? I think a good example is the recent debate over hydroxychloroquine. Some really low-quality observational data initially suggested that there may be benefits to individuals who have COVID-19 from receiving hydroxychloroquine. Subsequently, other papers suggested that there actually may be harm to patients.
It's not surprising that you would find harm in an observational study — just think about which types of patients with COVID-19 are going to be offered hydroxychloroquine. It's going to be the sicker ones. So, you could look at observational data and conclude that patients who get hydroxychloroquine are harmed by it when, in fact, what's really happening is that the people who are sicker tend to get the drug.
In those kinds of settings, where it's hard to do a randomized trial because the timing of the information is particularly important, there may be good ways to design natural experiments. So, you could, for example, look and see what happens in hydroxychloroquine use in the days after President Trump made announcements about it.
Suppose you observe that hydroxychloroquine use goes up in the days after the announcement. If we assume that the average severity of coronavirus infection in individuals is similar in the days before versus the days after that announcement, you could look and see what the average differences in outcomes are between those two groups. And you might, then, attribute it to the difference being usage of hydroxychloroquine, as opposed to patients being of differential illness severity or receiving other types of treatments.
My instinct is that looking across countries is going to be challenging because so many things are different across countries in health care systems, even aside from availability of testing and adherence to policies related to social distancing.
An interesting technique that people in our field use is called a border strategy. They look at counties that are along a state border. The idea is that if you live along a state border, you're probably subject to the same local environmental conditions, like people are using the same restaurants, they're going to the same stores, the culture of the people may be very similar.
But in the US, if a state passes something, it affects the people in that state. It doesn't affect the people just on the other side of the border. And so you can look at people who live along borders and focus your analysis on the effect of different state policies on people who are at the border, compared to people who live in an adjacent state on the other side of the border who were not affected by that particular state's policy. So that is one way to try to get around this problem that states may be implementing policies in times of greater need, or anticipated need or anticipated rise in cases..
Click here to see the full webinar, “Natural Experiments in Health Care: What Really Works,” which delves more deeply into this topic.
Continue the conversation on Twitter by connecting with us @ExecEd or with Dr. Jena @AnupamBJena
--Francesca Coltrera