On Making Cost-Benefit Useful

How to Help Policymakers Care About your Results

In this essay, I will propose a relatively novel way of thinking about analyzing program success as an input to policymaking. My take is that the answers provided by conventional program evaluation and policy analysis are inadequate in developing research findings that policymakers can act on. In the best-case scenario, research develops answers that are valid and reliable, but that doesn’t make those results intrinsically useful. To be useful, research has to ask and answer the questions that those with the power of the purse are asking. Lots of important, interesting, rigorous research falls short of this threshold. And thus, social science research is often not taken as seriously in policymaking as it might be.

The specific question policymakers are asking is, with limited dollars and too many needs, how do I choose where to invest?

More often than not, researchers are developing the information needed to answer that question. But researchers typically set aside these useful findings on their way to bigger and better outcomes. As Robert Pirsig notes, sometimes you are looking so hard for the truth that “the truth knocks on the door and you say, 'Go away, I'm looking for the truth,' and so it goes away."

The answer lies in thinking about where research results intersect with policymakers’ needs. I think that interaction occurs rather earlier in the research process than some of us might imagine.

Source: I asked ChatGPT for a little help with an image, and we came up with this, which GPT titled, Sack of Fortune at the Crossroads.

How CBA adds value

The purpose of a cost–benefit analysis is to attempt to aggregate all of the effects of changes in incarceration and measure them in a single unit: dollars. This allows the policymaker to concretely identify the expected impact of any policy change and consider how different policies might yield superior net benefits. Abrams, David S. "The imprisoner's dilemma: A cost-benefit approach to incarceration." Iowa L. Rev. 98 (2012): 905.

I have been doing work in and around the field of cost-benefit analysis for 25 years. I find it to be a handy tool, especially for policymaking. The problem with conventional program evaluations and policy analyses is that each study has a different unit of analysis, and the results are specific to the population under study. Such and such program reduces this and that bad outcome by some amount. That’s an excellent starting point for a discussion. But while it is necessary, in the language of social science, it is certainly not sufficient for policymakers to make hard choices about allocating scarce resources.

All decision-making, and this includes policymaking, is about choices. All choices involve comparisons. Choosing between two (or more) policies and programs requires that they be comparable, which requires that the bottom line is measured in the same way. If you say that one program improves 8th-grade algebra test scores and another program reduces truancy among 8^th graders, and you can only afford to fund one, how can you choose? You would want to know how valuable each of those outcomes is and the magnitude of the overall effect. And how much does it cost?

That’s a lot to ask for, but all of these elements are necessary to get to the bottom line and make a decision. If you wrestled with what to do with your kids over the summer, for example, you’ve gone through exactly this calculus.

These are hardly novel ideas, and these kinds of questions have motivated generations of cost-benefit analysis. As the Abrams quote describes, the goal of a CBA is to translate the many and varied outcomes of program evaluation and policy analysis into a single common measure—dollars. So, in the example above, about the 8^th graders, two CBAs would be undertaken. Analysts would figure out what it cost to implement the program that improved test scores and what the benefits were (in dollars). Analysts would do the same on the truancy reduction program. And then you could make a choice.

I like this example a lot because I don’t have an a priori belief that one of these is superior to the other. That makes it an especially compelling opportunity for research and evidence to contribute to the debate, since there isn’t a strong emotional attachment to one answer or another.

Going to school (not being truant) is obviously a precondition for learning, but if school is chaotic or the pedagogy is weak, just going to school is not going to solve a lot of problems. It’s a strong case to fund the truancy program, but not an ironclad one. Moreover, truancy is often not just skipping school to hang out or make trouble; it is often part of the complex challenges faced by poor people and poor schools. It can be a costly and wide-ranging problem to solve.

Improving 8th-grade algebra scores, by contrast, seems like a curiously specific goal. The research is really compelling, as completing 8th-grade algebra unlocks a whole host of workforce opportunities that are otherwise blocked. Any number of school districts—and the state of California—have jumped on this and implemented Algebra for All. But it turns out there are some counterintuitive challenges.

Anyway, this strikes me as the perfect example of a policy challenge where rigorous evidence could be really useful. We definitely want to do both of these things—keep young people from being locked out of valuable workforce opportunities and keep kids engaged in school for as long as possible. But there are real challenges to doing either one (which is part of why we haven’t solved these problems in the past) and taxpayers are leery of spending money on social programs unless their success is pretty certain (which is another reason we haven’t solved the problem, and there are other reasons of course).

Plus, it is a great opportunity to use cost-benefit analysis to level the analytic playing field. We can isolate the costs of each program and estimate the benefits in a common unit of analysis (dollars) so that policymakers and the public can make an informed decision.

Except that cost-benefit analysis is going to hit a brick wall that will greatly limit its usefulness (which is another reason we haven’t solved these problems).

In Search of Practical Answers, We Find Only Theoretical Ones

My headline gives away the answer to why cost-benefit analysis only gets you so far. Benefits from most programs—whether they are prevention or intervention—cannot be captured. The outcomes are real, but the savings are theoretical.

Let’s take each of these problems and think about what the benefits are. For the foundation of this discussion, let me borrow from a cost-benefit analysis of a student achievement program. I want to be clear that I am not criticizing this paper—the paper uses a conventional approach and is precisely how I would approach this question using a CBA. It is the convention I want to challenge.

The paper articulates some benefits of developmental education. Those benefits categories are:

• Better Health

• Reduced Crime and Social Costs

• Greater Productivity

• Additional Earnings

I think these categories are self-explanatory and equally appropriate for both programs. If you are truant less often or you complete 8th-grade algebra, it is reasonable to think that you may experience these kinds of outcomes. Or, at least, it is sensible to test and see if you experience these outcomes.

Now, here is the key idea. From the perspective of the public taxpayer or the policymaker, almost all of these monetized benefits of these outcomes are theoretical, not practical. When I say this, I feel as though I am standing on the edge of a precipice, ready to fall into an abyss of explanation. So let me use a simple example to illustrate.

Suppose there is an 18-year-old senior who has been chronically truant and receives a truancy intervention and starts attending school regularly. Suppose also that a careful statistical analysis reveals that the average 8-year-old with our students’ profile would have been expected to be arrested and incarcerated for a short stay in jail. But, because they returned to school through the truancy prevention program, they avoided all those bad outcomes. Our cost-benefit analysis shows that jail is really expensive, so we report that we have saved thousands of dollars by avoiding that outcome.

Here's the thing. Those dollar savings are both legitimate and wholly theoretical. It is accurate to say that those jail experiences were avoided, as were the significant costs associated with them. But no actual savings accrue to the system. No dollars were freed up in the correctional systems budget to be used for training or education, or to be returned to the taxpayer, because our student did not go to jail. This is mainly because all of these jail costs are sunk costs—once resources are appropriated to the system, they are gone, for good.

One way to think about it is that prosecutors and courts likely have someone else queued up they would like to send to jail, but could not, but now can. Florida International University scholars Stewart D’Alessio and Lisa Stolzenberg refer to this as a corollary of Parkinson’s Law—just like work expands to fill the time allotted to it, so too will more people be sentenced to jail if jail capacity increases.

This is true for most of the benefits. The health care savings either go in the pocket of the person who received the services (because they are healthier and don’t need medical services) or giant systems like Medicaid, where these changes do not affect total expenditures. My colleagues and I did a back-of-the-envelope calculation that if there were no fatal or nonfatal shootings in New York City, the overall effect on the health care system would be so small that it would not affect capitation rates, and there would be no savings to the government.

Now, you are probably thinking, so what? That’s no reason not to try to prevent fatal and nonfatal shootings! And of course, you are right. But to the policymaker whose job it is to allocate scarce resources, it matters that the savings are theoretical and not practical.

A Solution—Proximal Outcomes

So now we reach the end, where I offer a solution. I think the answer to this quandary lies in the findings of evaluations and policy analyses that are typically generated as part of the study process but that rarely make their way into topline findings. Specifically, most studies generate some findings about interim outcomes, such as whether people completed the algebra program or the truancy prevention program. Public health researchers refer to these intermediate outcomes as proximal outcomes. That is, there are the main effects of the program that occur sometime in the future (distal) and some other interim outcomes that occur right now (proximal).

I am arguing here that the distal outcomes, at least in the cost-benefit framework policymakers prefer, are too theoretical and not practical enough. What they want are proximal outcomes that tangibly free up resources in the short term that can be redirected for other uses. Another way to say that is, what policymakers really want are programs that pay for themselves.

When I wear my cost-benefit analyst hat (it’s a tricorn), I often get asked what program I would recommend doing first. My answer is, whatever program has been empirically demonstrated to be successful locally that pays for itself. Sustainability is the hot buzzword in social science research. Want a sustainable program? Find a way for your programs to pay for themselves. And not just in theory, in a practical way, where you can actually point to where the savings are in the budget.

And that’s the key. Focusing on intermediate (proximal) outcomes in the process of doing a cost-benefit analysis is a way to identify program benefits that a researcher can actually point to in a budget and say, your savings from this program are right there.

I think there is likely to be a set of candidate proximal outcomes in every evaluation of a government program. Let me just give you an example and we can wrap this up.

Let’s take as our example, the PAX Good Behavior Game (GBG). They describe it this way:

The PAX Good Behavior Game® is an evidence-based universal preventive intervention applied by teachers in the classroom. This evidence-based practice consists of a set of research-based strategies with origins in behavioral science and neuroscience that operate together to improve children’s self-regulation. Teachers implement these strategies as part of their daily routines in carrying out tasks such as getting students’ attention, selecting students for tasks, transitioning from one task to the next, working as part of a team, limiting problematic behavior, and reinforcing pro-social behavior.

So, it’s a classroom intervention designed to incentivize cooperative behavior and limit disruptive behavior. Suppose we did a cost-benefit analysis of the Good Behavior Game. We would likely end up with the same set of outcomes described above: in the future, are kids who participate in BGB healthier, do they stay out of the justice system, have better pay, etc. But none of that appears in the budget, or at least not soon.

But along the way, the study probably produced some intermediate (proximal) outcomes that tested whether GBG did the things it was supposed to do. Was it successful, “getting students’ attention, selecting students for tasks, transitioning from one task to the next, working as part of a team, limiting problematic behavior, and reinforcing pro-social behavior”?

Now, all of these outcomes are all part of the same latent construct: a peaceful, cooperative classroom. So, what are the broader implications of a peaceful, cooperative classroom? It’s a nice place to work. What happens when people go to a nice place to work? They want to work there, so they don’t quit.

It turns out that the teacher turnover is a huge problem in many school systems, and that school systems spend big bucks on teacher retention, training/onboarding, and recruitment. But if teachers are happier in the classroom, you can avoid paying those expenses. And that’s a particular line in the budget you can use to demonstrate your savings. There’s even a dashboard that will do the calculations for you.

I don’t think there is anything unique about the Good Behavior Game that lends itself to this kind of analysis. In fact, I think school settings are relatively harder environments to find practical savings—workforce and health strike me as much easier.

The problem is not that practical savings from short-term outcomes of effective long-term prevention is hard to find, the problem is that we are not looking.

And that’s my concluding remark: if you do good things down the road, you are probably also doing a lot of good along the way. Let’s monetize those outcomes too, because that’s the information policymakers are looking for.

Musical Interlude