The Gold Standard in Research Design

A few days ago a piece ran in the New York Times criticizing the methodological approach the Center for Medicare and Medicaid Innovation Center uses to assess the effectiveness of different health care delivery options.

I recently moved from the social sciences (“International Relations” but with a focus on the socio-economic causes of conflict…technically a “comparativist” field or, because my research focused on Mexico, Venezuela, Peru, and Colombia…an “area studies” field). Now I work in behavioral health care and public health. In truth, I have always worked in policy. In studying policy from an academic point of view, we hear tell of these fancy methods known as Randomized Controlled Trials, but we rarely get a chance to read of one in our field. Dani Rodrick, a developmental economist preaches their use and is developing ways to utilize them in the field, but apart from him and, given our respective limitations as finite creatures in a world of scarcity, most of us never get a chance to actually conduct one.

Although I have always worked in policy, this is my first experience with health care policy. Most of my hallway mates are doctors. Some of them are medicals doctors, most of them are PhDs in psychology or PsyDs (Doctors of [Clinical] Psychology). Many of my current coworkers are from the world of randomized controlled trials, which they refer to as “clinical trials.” As a professional research assistant, I have two main functions: translating the findings from clinical trials for policy-focused folks, and translating policy stuff for clinical trial-focused folks. So when Austin Frakt and Adrianna McIntyre write “And just because health policy is closer in proximity to medicine (and its many RCTs) doesn’t actually make health policy more amenable to this kind of study than any other policy domain,” I feel them.

When I tell people my background, they look at me like I’m crazy. But this really is policy work. This really is outside the lab. Randomized controlled trials are great but they have limitations, extreme limitations.

Prior to this, in social sciences research, the biggest methodological conflict was always the quantitative folks versus the qualitative folks, the large-n folks versus the case study folks. My own thesis attempts to straddle that line, using a case study to fine tune certain macroeconomic models. It’s an uncomfortable world in-between these two approaches. So, trying to act as mediator between policy and RCTs now is uncomfortable–but familiar–territory.

So what am I trying to say in this long-winded comment? Basically this: In the world of lab coats, pills, placebos etc. the RCT is the “gold standard” of research designs. But outside that world, they are flawed and limited. And…for that matter, not all RCTs are equal. Many of them are flawed in their own ways, in their design, in their implementation.

But it’s this idea, specific to the hard sciences, that RCTs are “the gold standard” that I think is to blame for that NY Times “hit piece.” No one else outside the laboratory world thinks that RCTs are “the gold standard.” RCTs, like any method, are good at what they’re good at, and bad at what they’re bad at. They are a method, not the gold standard. If we dropped this idea, then it never would have occurred to anybody to critique the Innovation Center for not using RCTs when RCTs are either impossible or the wrong method entirely.