Perhaps there is something unique about the people studied by RCTs in the criminal legal space that could limit generalizability. For instance, the people in these studies often come from marginalized groups; they have little access to society’s wealth and resources. Do these factors make their life trajectories particularly difficult to change? Would interventions made to the lives of better- resourced individuals make more of a difference?
At first glance, those who occupy more privileged places in society may appear to have a greater capacity to respond to interventions. They have greater access to resources to help them advance; their higher levels of education may give them enhanced knowledge about opportunities and consequences. But to the extent that more advantaged groups have a greater capacity to fulfill their wishes, they can accomplish this on their own. The equilibrium state of advantaged groups looks different from the equilibrium state of less advantaged groups. But, like everyone else, they are living the best life they can create for themselves given the opportunities provided to them, the knowledge they have, and the set of skills and burdens they carry. If it was easy for them to have made a meaningful improvement, they would have done so already.
The relevant question here is the extent to which externally imposed and short-term changes to circumstances—interventions—lead to meaningful, lasting, and replicable changes in people’s life trajectories. I don’t see why this would be more likely with an advantaged population than a disadvantaged one. To the extent that advantaged groups have greater capacity, they are starting from an equilibrium in which they are closer to having achieved their goals. To the extent that their goals remain out of reach, that’s due to the constraints relevant to them. And there is no clear reason to assume that these constraints are more easily cleared away than those relevant to less advantaged populations.
One could also speculate that the claim applies only to the set of interventions relevant to criminal justice reform. Again, I don’t think this is the case. For practical purposes, I’ve limited my empirical analysis to one substantive area, but I believe that, at least in broad strokes, the claim extends beyond this setting.
My beliefs are informed by my knowledge of the empirical literature in other fields. I was trained by development economists and I frequently attend conferences and collaborate with economists studying education, health, and labor. In these fields also, interventions evaluated via RCT rarely find large or lasting benefits. Microcredit (loaning small amounts of money, often to women) was, for many years, the darling of the development world. Eventually, it was shown to have little to no net benefit in most places. Health insurance, randomly allocated via lottery, was shown to increase healthcare usage and reduce bills sent to a collections agency. Yet, it had no statistically significant effect on physical health or labor market outcomes. The fact that most interventions in the social world have little impact is so ubiquitous that it has been dubbed the Iron Law of Evaluation: “The expected value of any net impact assessment of any large-scale social program is zero.”
To the extent that RCTs in other fields may find effects more frequently, this could be partly because they are more interested in quantifying a direct effect rather than establishing an indirect or longer-term effect. For instance, a group of large-scale RCTs conducted in the 1960s-1980s evaluated the impact of providing a guaranteed income to low-income individuals. One of the primary goals of these studies was to quantify how much less people would work, with few questioning whether there would be a labor supply effect at all. As expected, all studies found that cash transfers led to lower labor supply. But when it came to more indirect outcomes—health, consumption habits, etc.—the researchers found that, overall, “the lives of recipients were not altered dramatically by the payments offered in the experiments.”
In sum, I expect my claim about the structure of the social world applies to many different types of people and many different types of interventions—at least where the proposed mechanism of causal influence is indirect or convoluted. The more speculative it sounds, the less likely it is there will be a robust and replicable causal relationship.
For those who desire larger-scale change, and for whom incrementalism (with interventions constrained enough that they can be evaluated with RCTs) is too slow and uncertain, there really remains only one option: systemic reform. This Article shows that limited-scope, isolable interventions rarely lead to meaningful change. Those who desire meaningful change must therefore seek interventions outside the scope of what is evaluable via RCT. This includes changes that are so multipronged and entangled that it is impossible to hold all else constant. This also includes changes that are so large in scope that experimental evaluation is infeasible.
Systemic reform is not an “intervention” in the way I’ve been using that term here. It’s not something you can do on your own; it requires changing the hearts and minds of large numbers of people, as well as changing the concrete structural factors of our lived experience. It’s hard to know what systemic reform will bring, not only because we cannot test its impact empirically, but because it’s very hard to imagine a world that is otherwise the same as ours, while also being deeply, structurally different. When it comes to systemic reform, we are flying half-blindMegan T Stevenson – Cause, Effect and the Structure of the Social World – Boston University Law Review