Skip to content

Cascades, tides and shifting stars in social interventions: Megan Stevenson on cause, effect and the limits of RCTs

Recommended.

A wide-lens view on more than fifty years of RCTs in the criminal legal space reveals a few common themes: most interventions don’t work, and the ones that do tend not to replicate well in other settings. While this may be disappointing from the perspective of learning how to engineer social change, it teaches us something valuable about the structure of the social world. Namely, it teaches us that the social world is full of what I call “stabilizers” and short on what I refer to as “cascades.” Note that I am not trying to explain my empirical claim by introducing these concepts. Rather, I am simply restating it in more abstract terms. Language has power, and, for me, this abstraction has helped me see the world in new ways.

Stabilizers are the set of socioeconomic forces that resist externally-imposed change. Imagine an orange in a large bowl. One can push the orange up the side of the bowl, but as soon as you let go, it rolls back to the bottom. A job-training program may provide a temporary job as well as some job-relevant skills. But whatever socioeconomic forces made it hard for that individual to find a job in the first place—for example, a society in which access to opportunity is deeply segregated—prove to be powerful inhibitors. After the program is over, the participant returns to the place they would have been absent the intervention.

The orange analogy has some strengths, but it implies a return to stasis and a direct force-counterforce dynamic. Stabilizing forces don’t necessarily need to embody either. Consider another analogy: the tides. When the tide is pulling out to sea, the flotsam and jetsam will be carried along with it. A gentle cross breeze might create some eddies but will have little impact on the overall direction of flow. If change is governed by widescale social forces, then the interventions evaluated via RCT might be like this gentle breeze: ultimately irrelevant.

Stabilizers are closely related to a type of social change that I refer to as “cascades.” Cascades are forces that magnify small changes, that turn a small intervention into a large and lasting effect. Consider, again, the example of the job-training program. Theoretically, a job-training program could launch a cascade. It could help recently released prisoners secure employment. Employment could then help secure housing, which could then create independence and security, which could in turn prevent drug use and other unhealthy behaviors, and so on and so forth until the person is reintegrated as a thriving member of society.

A cascade is defined by the idea that a small but well-timed intervention leads to a cycle of change that accumulates over time and affects many areas of one’s life. Cascade narratives can be very compelling. But that doesn’t mean they are true. RCTs teach us that very few interventions launch such a virtuous cycle of accumulating benefits. The social scientists’ holy grail—the small, inexpensive intervention with large, widespread, and lasting gains—appears to be mostly myth.

Occasionally, however, someone claims to have found such a holy grail. Some intervention is demonstrated to be highly successful in one setting: so much so that other jurisdictions try to mimic their success. Unfortunately, the process identified in one setting rarely ports well to others. Such instances suggest that causal processes are highly reliant on specific contextual conditions. As a program expands, it may have trouble recruiting and training staff to the same skill level. The original success could be due to the charisma of a key figure, without whom the intervention has much less impact. Or it could be contingent on a particular set of background conditions—low crime rates, low unemployment rates, etc. If success relies on a particular alignment of the stars, a causal process detected in one setting teaches us little about what will happen once the stars shift.

Megan T StevensonCause, Effect and the Structure of the Social World – Boston University Law Review

I'd love to hear your thoughts and recommended resources...