How to make theories of change more useful?
A fair amount has been written recently questioning the value added of theories of change. Have we gone through a hype cycle? Are they just a tickbox ‘logframe on steroids?’ Are they still useful?
As Thomas Dunmore Rodriguez explains:
“The end result can often remain a fairly sketchy story of change, with lots of untested assumptions.”
Dunmore Rodriguez recalls a complex theory of change mapped across the concentric circles of a socio-ecological model, illustrated below:
Why do we so often end up with a fairly sketchy story of change with lots of untested assumptions?
In my view, despite the very real restrictions imposed by the pandemic, the limitations Dunmore Rodriguez identifies have as much to do with organisational politics as they do with any particular tool, method, or process employed.
A theory of change is a hypothesis of how and why change happens, but this is rarely what a theory of change is in practice. Theories of change tend to have very little explicit theory, and hypotheses are often untestable or untested. Rather than a programme theory of how and why change happens, theories of change are commonly used as a process for organisations to agree upon how to translate part of their mission into a public strategy.
Theories of change often get rolled into “strategic planning” or a “strategy refresh.” If you’re trying to capture the collective ambitions of a group, those ambitions are legion. Particularly in any large organisation (like my previous employer, CARE), strategic plans are more about a restatement of mission and normative goals, and first and foremost a rebranding exercise.
Theories of change as political settlements?
Theories of change often become casualties to the fissures in the political settlements within an organisation. At organisational level, theories of change are ultimately manifestations of such a settlement:
‘A common understanding between elites… bargaining outcomes among contending elites.’
They are “deals” about the goals publicly expressed by an organisation, achieved through bargaining among directors, heads of teams, etc. It’s commonly a “representation” exercise, but it’s a representation exercise for elites within the organisation. And those elites are most commonly in London, or Washington, or wherever headquarters is (i.e. where the money is).
What enters the diagram at this level is rarely about a theory (a hypothesis), and the evidence base underpinning such a hypothesis (or hypotheses), the reasonableness of assumptions or sensitivity to context all evaporate in that bargaining process. So, it’s hardly surprising we’re passing the peak of inflated expectations. The trough of disillusionment comes, in large part, from our exhaustion with arduous, unenlightening, and non-inclusive strategic planning processes. For all those going through a strategy refresh at the moment, my sympathies are with you.
Define clearer boundaries
At the level of a programme or a project, how we reach this common understanding is somewhat different. This is where today’s blog from Dunmore Rodriguez is more relevant. However, we still find some of the same issues. The diagram above from Brazil is a theory of change in the most expansive sense. It’s close to a theory of everything. In this respect, it’s more of a vision statement of aspirations, but not necessarily very useful as a compass or a strategy, let alone amenable to a log frame or a results framework.
I think we’ve become rather distracted by the allure of systems theory, and yet rather misunderstood it. One of the most important aspects of systems theory is drawing boundaries.
As Duncan Green mentions, “many organisations use the term ‘Theory of Change’ to describe their Theory of Action” — their organisational strategies and tactics, rather than their hypotheses about how change happens. There is an important difference between the two, but I fear there has been an overcorrection where many organisations aspirations to go bigger are no longer able to separate the wood from the trees. As one of the authors of the Outcome Mapping guide put it:
“Stop trying to change the world; focus on your sphere of influence.”
What emerges from the theory-of-everything problem is a reminder that many theories of change have issues of mission-creep. There’s a certain hubris to a lot of theories of change which we don’t always acknowledge. This often emerges from the promises organisations make to donors (to acquire and legitimise funding), as well as the lofty (and unrealistic) ambitions of organisations themselves.
At the end of the day, your intervention, your programme, your organisation are all a drop in the ocean. You may well have ripple effects, but a lot of talk of “transformation,” “impact” (and even “super-impact”) are more marketing than they are strategy (much less testable hypotheses).
As Toby Lowe recently explained with the example of an obesity system map, “only four of the 108 factors which go to make up the outcome of obesity are directly related to public service delivery.” So, if you thought your service delivery intervention was going to transform obesity, think again. We need greater humility, but also the courage to draw and explain the boundaries of our influence.
Be more problem-driven
Being “problem-driven” is yet another buzzword I hear (and use) frequently. I’m not necessarily advocating that all organisations should employ the Problem-driven Iterative Adaptation toolkit, but we should all be considering more carefully where within the system our organisations have genuine potential value added, and “leverage points” to make a bigger difference (I know this is in Oxfam’s own guidance somewhere).
One key problem with political economy analysis, or any form of context analysis, is that it’s often context that doesn’t have a clear and direct relationship with the proposed intervention. It’s very common for organisations to list contextual factors they can do nothing about and have no strategy to address (or mitigate the effects) and list stakeholders they never mean to engage. Last year I argued that there’s some promise in stakeholder-driven approaches to political economy analysis, but you have to be clear about what problems (or cluster of problems) you and your partners can reasonably influence before your can identify relevant stakeholders to engage in that process anyway.
Be more evidence-based
There is more than one type of theory-driven approach. Guides to theory-based evaluation distinguish research-based and tacit (or stakeholder-based) theories of change. In the development sector, we seem to have completely forgotten that the quality of theory is predicated on the quality of evidence underpinning that theory and the reasonableness of the assumptions we have where we lack evidence.
We are inevitably making bets about what will work under conditions of uncertainty, but it’s not as if our ideas are so unique that no-one has tried any of them before. Sue Funnell and Patricia Rogers’ seminal work on theories of change even outlines what they call “archetype” theories of change (or meta-theory) — (1) information, (2) community capacity building, (3) case management, (4) direct service delivery, and (5) carrots and sticks. Just like there are only seven basic plots in stories, there are only so many common theories.
So, before you enter the lions den of the theory of change workshop, remember we stand on the shoulders of giants, and don’t just rely on the collective wisdom of those in the room. While Dunmore Rodriguez is certainly right that theories of change are best developed in a workshop setting (for both political and technical reasons), going online may provide greater time and (head)space to reappraise the evidence base upon which our tacit theories are based. And if you have only the 2-hour window Dunmore Rodriguez mentions, you can prepare better to make the most of it.
Be more explicit about your assumptions, and test them
The same exercise of organisational pageantry and isomorphic mimicry engenders another critical flaw which Dunmore Rodriguez identifies — assumptions. However, in order to be more “much more scrupulous in testing the assumptions we make ourselves” we first have to be explicit about what those assumptions are and explain what type of assumptions these are. Oxfam’s own Irene Guijt has an excellent short paper on this. I can’t help but ask: why has this been forgotten within Oxfam?
Theories of change are a form of causal diagram. Arrows are supposed to represent that supposed causality. What is so often frustrating about developing theories of change is that we spend so much time focused on developing grand statements of change and we spend so little time thinking about the arrows and the assumptions which should sit beneath those arrows. I’ve explained previously why we should think of theories of change being primarily about assumptions. As Huey Chen put it:
“A theory-driven approach requires [us] to understand assumptions made by stakeholders… when they develop and implement an intervention program[me]… Based on their program[me] theory [we] systematically examine how these assumptions operate in the real world (Chen, 2015: 25; see also Weiss, 1995; Donaldson, 2008; Funnell and Rogers, 2011).”
Yet, rarely do we systematically examine or monitor our assumptions, let alone update these. And perhaps most importantly for an organisation like Oxfam, a lot of assumptions we make aren’t causal assumptions but normative assumptions (the values and norms the organisation itself espouses). If what is represented in the diagram is a set of moral precepts, we’re not really looking to “test” these. However, for causal or implementation assumptions, there are plenty of good tools out there which can help us.
Indeed, as Dunmore Rodriguez explains, one key part of challenging our assumptions is bringing diverse participants into the room to share their different perspectives and to get views from outside our own bubbles. One one hand, this is about “ground-truthing” the strategy (and causal pathways) designed in central office, but it can be just as important to consult stakeholders you know disagree with you. Our aim shouldn’t be simply to add more aspirations (the dance between vision and mission), but to self-critically interrogate whether our hypotheses are valid. While the pandemic doubtless limits our capacity for ground-truthing, it does provide the opportunity to invite more critical voices from our peers or those we mean to influence. Zoom can help you zoom out.
Be more iterative, but focus
Dunmore Rodriguez further notes that current uncertainties mean that theories of change are “likely to need updating almost immediately, yet this doesn’t seem to be common practice.”
You don’t necessarily need to update all of your theory of change, especially not if it’s a “narrative” or “overview” theory of change (see here for a discussion). Dunmore Rodriguez’s concern, I fear, is a symptom which emerges from the mission creep problem I mentioned above. If, instead of worrying about the complex interactions across the system as a whole, you focus your area of enquiry into a part of that system and whether your assumptions are likely to hold in relation to that, then you can zoom in on why those particular assumptions might merit revision, and thus what implications that may have for double or triple loop learning.
You’re not likely to adapt your strategy if you never seriously question whether the boundaries you establish are reasonable, whether the norms and values you espouse are sound, and whether the pathways you propose are actually credible. The whole point of any exercise you choose should be asking yourself difficult questions. The specific widgets we have are secondary.
A 2-hour shallow dive workshop only gets you so far, and I’m not convinced human centred design parachuted in from Palo Alto necessarily gets us any further. However, I do think that there are opportunities to combine theories of change with scenario planning. In particular, context monitoring and assumption monitoring are intimately connected. So, there’s an area for further exploration.
