Windows on the world: The power of assumptions in uncertain times

In my last blog on theory-based Monitoring Evaluation and Learning (MEL), I explained why relationships matter, and how to assess change realistically and incrementally. And in the blog before that, I explained how you can assess the quality of evidence. In this blog, I want to focus on the assumptions which underpin connections in your logic.

Strong theories of change

Nearly a decade ago, the Stern review showed us that theory-based methods have a different approach to causation. We’re not looking at the frequency of association between cause and effect (as in statistics) or at counterfactuals (as in experiments), but at either combinations of causes that lead to an effect (configurational explanations) or identifying causal mechanisms (theory-based explanations). When we’re looking at theories of change, we’re primarily looking at mechanisms. Rather than asking whether a programme works or not, we’re asking what it is about the programme that makes it work (Pawson and Tilly, 1997: 66). We’re not simply making if… then statements, but rather if, then… because statements.

I want to convince you that theories of change are chiefly about assumptions. Theory-based evaluation guru Huey Chen describes a theory of change as:

‘A way to describe a set of assumptions that explain the mini-steps that lead to the long-term goal and the connections between program[me] activities and outcomes that occur at each step on the way. It is what must be done [by different actors] to achieve the desirable goals (Chen, 2015: 66).’

Still not convinced? Chen elaborates that:

‘A theory-driven approach requires [us] to understand assumptions made by stakeholders… when they develop and implement an intervention program[me]… Based on their program[me] theory [we] systematically examine how these assumptions operate in the real world (Chen, 2015: 25; see also Weiss, 1995; Donaldson, 2008; Funnell and Rogers, 2011).’

If you take this on board, then theories of change are less about a chain of boxes from activities to impact and more about describing and explaining our assumptions in between or underneath the boxes.

Lovely Dhillon and Sara Vaca ( 2018) point out that strong theories of change also include causal links, mechanisms, and assumptions (see layers of a theory of change diagram). These allow for a deeper understanding.

What are assumptions?

If assumptions are so important, what actually are they?

Assumptions are (generally unspoken) beliefs about what is true or expectations of what will happen. They are tacit assertions about the world which we rarely question or check. They stem from and represent values, norms, and ideological perspectives which inform our interpretation and understanding of reality and how the world works (see Guijt, 2013; Van Es et al. 2015; Morrell, 2018).

As critical thinking guru Stephen Brookfield points out:

‘Assumptions are rarely right or wrong, they are contextually appropriate (Brookfield, 2011: 21).’

Through this lens, theory-based methods to MEL are about what we believe is true in a particular context at a particular moment in time rather than true everywhere, forever. As the writer Isaac Assimov once advised:

‘Your assumptions are your windows to the world. Scrub them off once in a while, or the light won’t come in.’

Why do assumptions matter?

‘Assumption hunting is at the core of critical thinking (Brookfield, 2011: 197).’

‘Unexamined and unjustified assumptions are the Achilles heel of development program[ms] (Archibald et al. 2016: 1).’

Assumptions are also a means to bring in complexity thinking to help us gauge the degree to which we understand a system and the various relationships, interactions, and interdependencies within it. Whether your assumptions hold true over the next few months with COVID-19 may actually keep you and your loved ones alive.

If assumptions are central to critical thinking (including evaluative thinking) and failing to examine or justify our assumptions can potentially derail development programmes (and even put you in harm’s way), why do we spend such limited time thinking or talking about them?

One curse of the log frame is that assumptions are the box on the right hand side, so if we read and plan from left to right (as we do in English, rather than Arabic) we think about them last, commonly when we’re already tired and just want to go home after a long day at the office. As such, they’re often just a parking lot for a long list of self‐evident, usually shallow, statements (Guijt, 2013, see also Davies, 2018). Assumptions are typically written in the affirmative (“girls will turn up to class,” “the sun will rise tomorrow”), but in the assumptions box we also find a litany of external risks written in negative like: “resources won’t be cut (when you actually suspect they might quite soon),” “staff rotation won’t disrupt operations (when you actually anticipate staff rotation in the near future),” “elections won’t affect the project (when you know there is a contentious election on the horizon),” even, “a pandemic won’t break out (but, here we are).” The fundamental difference between assumptions and risks is just how you plan to address them coming true or not.

Prioritise your assumptions

You can list dozens of these statements, so you should prioritise those that you consider most important (and relevant). Building on complexity theory, you can help define which assumptions are important based on how certain (or confident) you are about them holding true and whether they are likely to have a large influence over your intervention. Assumptions that are highly uncertain but have a high level of influence over your intervention are mission critical.

Here are three criteria you might consider in identifying an important outcome in your theory of change:

• It is a central storyline for the initiative (your story of how change is supposed to happen);
• Lots of arrows = main channel/pathway (a lot of outputs lead to this outcome);
• Lots of actors have a stake (positive or negative) in the materialisation of the outcome.

You should then think about which key assumptions might be weakest (i.e. which are most vulnerable).

Vulnerability criteria might include:

• Limited evidence of what works in this context (need for further investigation);
• Low level of agreement on stakeholders’ preferences or what will work (diverse perspectives);
• Highly exposed to external threats (e.g. political reshuffles, climate shocks).

See Patton, 2010 and Williams and Hummelbrunner, 2011 for some of my inspiration.

I hardly need to explain that COVID-19 is one such case where we still have limited evidence of what works in context (as it’s new), still strong disagreement on “the science” (especially the validity of some behavioural science and accuracy of mathematical models), and the strategy the UK government is pursuing may leave many of us highly exposed (i.e. it is a risky strategy).

Disaggregate your assumptions

However, there are various things you can do to help understand and manage this uncertainty. Brookfield ( 2011: 17–19) helpfully refers to three types of assumptions. Borrowing from Irene Guijt ( 2013), I recommend a fourth to underscore risks in how development interventions operate:

1) Paradigmatic assumptions: Structuring assumptions we use to order the world into fundamental categories. Most commonly, these are belief systems, world views, and philosophies. Patriarchy or feminism, free market capitalism or socialism, positivism or constructivism might all represent paradigmatic assumptions (see Kuhn, 2012 on paradigms). For COVID-19, this might include how the government and the general public view the relative validity of epidemiology, mathematical modeling, or behavioural science for weighting government decisions. These provide an actor’s or organisation’s underlying rationale.

2) Prescriptive assumptions: These are assumptions about what we think ought to be happening in a particular situation. Such assumptions are about how we believe we and other actors should behave. This is at a lower level than paradigms. It reflects what you believe to be the right (or wrong) course of action. These are also called “normative” ideas because they speak to social norms or other moral premises that shape what course of action is desirable (see Schmidt, 2010 on normative ideas). If your solution isn’t desirable, whether it works or not doesn’t really matter. For COVID-19, if the general public views the UK government’s strategy of “heard immunity” as unethical, even if citizens eventually comply, they may not forgive the government for allowing a higher mortality rate than might be achieved through an alternative (and less risky) strategy.

3) Causal assumptions: These are assumptions about how different parts of the world work and about the conditions under which these can be changed. These are the events and conditions needed (i.e. necessary) for the associated causal link to work-for the cause to lead to the effect. These are the most common form of assumptions. For COVID-19, for its strategy to work, the UK government must assume the general public will trust them, accept the science and logical connection to the chosen policy response, and follow their advice to keep calm and carry on (some highly vulnerable assumptions there).

4) Operational assumptions: These are assumptions about the operating environment for delivering a programme (Chen 2015 calls this “ecological context” as it can directly affect operations). These assumptions might include external context, such as issues of political stability, freedom of expression or movement, environmental factors (e.g. epidemics). These provide access and opportunities for different stakeholders. For COVID-19, there are relevant assumptions about how global supply chains function and how quarantine restricts movement (i.e. the great toilet paper crisis of 2020).

Test your assumptions

There are a great number of tools you can use to test your assumptions. Below are the five options I find most interesting:

Critical Conversation Protocol (CCP)

Brookfield ( 2011) recommends dozens of exercises. My favourite is a Critical Conversation Protocol (CCP). The CCP involves three roles: storyteller (the focus of the conversation), detectives (the assumption surfacers), and umpire (the monitor and arbiter). The storyteller tells a story about an event. The detectives ask questions of the storyteller and then describe the storyteller’s assumptions. They are encouraged to provided alternative interpretations of the events. Then, participants collectively conduct an experiential audit, including an appraisal of what participants have learned and an overall summary by the umpire. All of this happens in less than an hour.

The Five Whys

The Five Whys technique has its origins in the quality improvement efforts of the Toyota corporation.

It’s an exercise to assess what caused an end result and reflect on why that happened (Serrat, 2009). By asking “why” five times, one can usually peel away the layers of symptoms that hide the cause of a problem. Jonny Morrell ( 2018) explains how this can be used specifically for assumptions. It’s also used in Problem-driven Iterative Adaptation (PDIA). So, when a donor or evaluator tells you to use PDIA, I recommend you ask why at least five times. See the toolkit. This exercise can take minutes or hours.

Fishbone (or Ishikawa) Diagrams

Ishikawa diagrams are another exercise taken industrial engineering in Japan. These diagrams are also part of the PDIA toolkit, and designed as part of problem analysis. They aren’t much used in evaluation, but as Morrell notes ( 2018), they can also be used as a way to depict a theory of change and he argues it is particularly amenable to identifying assumptions. In its simplest form, the diagram consists of an arrow with an objective at the far right, and major requirements for success intersecting the arrow. You then break down each requirement into its own requirements. This can take minutes or hours.

Causal Link Monitoring (CLM)

Particularly if you have a logic model, you might consider Causal Link Monitoring (CLM). The great thing about CLM is that it forces you to look specifically at how assumptions link activities and outputs or outputs and outcomes. It does this by suggesting that implementers must use activities to produce outputs and actors must use outputs to achieve outcomes (Britt et al. 2017: 26). So, this simple logic focuses on response to stimulus, rather than simply the activities we control. For example, local research partners use (new) capacity to identify viable opportunities for climate change adaptations, or farmer organisations facilitate and promote producers’ use of new technologies and practices. This part of the method will typically take a few hours.

Assumption-based Planning (ABP)

Another great option is Assumption-based Planning (ABP). This is really well explained and adapted by Bob Williams and Richard Hummelbrunner (2011) in their brilliant book Systems Concepts in Action: A Practitioner’s Toolkit. To my mind, this is the only systems book I’ve read that is genuinely practical. It was developed by RAND corporation to help the US Army with its mid and long range planning. It’s found to be most effective in uncertain environments (i.e. war, pandemic). Here’s the original version. I’ve adapted this myself in the prioritising assumptions above. You identify important assumptions, then identify which of these are vulnerable, define signposts, then determine shaping actions and hedging actions. Shaping actions are those to help take control of uncertainty and help influence the external environment. Hedging actions are those to prepare for failure and cope with this internally. Shaping actions are thus about re-strategising; hedging actions are about re-planning. This will typically take a few hours. This is probably the best tool for the current crisis.

Hopefully, I’ve helped convince you to spend a little more time thinking about assumptions (especially given that over the next few months with COVID-19, some of your assumptions may be genuinely critical). If you have another 10 minutes, I recommend Irene Guijt’s forgotten Toc Reflection Notes 3: Working with Assumptions in a Theory of Change Process. It’s only five pages long, and well worth it.

In the next blog, I will write about more effectively narrating behaviour change.