Does qualitative evidence count in evidence-based budgeting?

By ·

Short answer: Yes, but not as proof of impact. Evidence frameworks keep their top ratings for comparison-group evaluations [S3]. Federal and UK guidance treats qualitative research as part of the evidence base for designing programmes, understanding implementation and explaining why results differ [S1][S4]. Use it for those jobs, and say plainly what it can't show.

1. Evidence ratings reward impact studies, and that's appropriate

Evidence-based budgeting asks a fair question: is there good reason to believe this programme works? Most state frameworks answer it with a ladder.

Tennessee's Evidence Framework has five steps: Logic Model, Outputs, Outcomes, Evidence and Strong Evidence [S3]. The top two require rigorous impact evaluations, such as randomised controlled trials or quasi-experimental designs with a comparison group [S3]. No amount of interview or story data moves a programme onto those steps, and it shouldn't.

That's the limit. The rest of this piece is about where qualitative evidence does count.

2. Federal guidance names qualitative methods as part of the evidence base

US Office of Management and Budget memo M-21-27, on agency evidence-building plans, lists the methods agencies can use, including "qualitative research, ethnography" and "well established processes for community engagement", alongside participatory, community-based and user-led research [S1].

The US Department of Labor's evidence glossary makes the same point at the level of study types: descriptive studies can use quantitative or qualitative methods [S2].

In other words, the federal evidence framework has room for qualitative work. The question is what job it does.

3. Qualitative evidence answers "why" and "under what conditions"

Impact evaluations tell you whether a programme changed an outcome on average. They often can't tell you why it worked in one place and not another, or what got in the way.

Tennessee's own Learning Agenda says its goal is to learn which programmes and policies best serve residents, "understand why they work, and under what conditions" [S3]. The UK Treasury's Magenta Book, the central guidance on evaluation in UK government, frames process evaluation around questions like "What worked well and less well, and why?", and says it uses both quantitative and qualitative methods [S4].

Those are the questions budget analysts ask when a programme's results are mixed, when it is being scaled, or when it is being redesigned.

4. Qualitative evidence has its own quality standards

"Qualitative" doesn't mean "anecdotal". The UK government's Quality in Qualitative Evaluation framework asks whether qualitative evidence is contributory, defensible, rigorous and credible, and sets out appraisal questions for each stage of a study [S5].

For a budget or evaluation office, that gives a practical test. Before relying on qualitative findings, ask:

  • Was the question clear and tied to a decision?
  • Who was heard, and who may be missing?
  • How were the accounts collected and interpreted, and by whom?
  • Can a reader trace each finding back to the evidence behind it?

Designing Sensemaking Studies sets a similar bar at the start of a study: "If participants cannot articulate the purpose in their own words, the study is not ready to launch." [BOOK, p.137]

5. Where it fits in a budget cycle

Qualitative work tends to be most useful at four points:

  1. Before the Logic Model step. Hearing from people a programme is meant to serve checks whether the logic model describes their real situation.
  2. At the Outputs and Outcomes steps. Process evaluation explains why participation or completion is lower than expected [S4].
  3. Alongside an impact study. Implementation findings explain why effects vary across sites or groups.
  4. For a learning agenda. Many learning-agenda questions are "why" and "how" questions that impact studies alone don't answer [S3].

6. How to present qualitative evidence to a budget office

  1. Say which step it supports. Label it as design, implementation or explanatory evidence, not impact evidence.
  2. Pair it with the numbers. Link each finding to the output or outcome measure it helps explain.
  3. Show coverage. Report who took part, through which channels, and who may be missing.
  4. Show the trail. Make it possible to go from a finding to the accounts behind it.
  5. State the limit. Note what an impact evaluation would still need to show.

Presented this way, qualitative evidence isn't competing with impact evaluation for a rating. It answers the questions the rating doesn't.

Sources