Ignorance, uncertainty, and the importance of judgement
I’ve read several books by the physicist Carlo Rovelli recently, and they’ve given me the opportunity to reflect on my ignorance. I have no expertise in physics at all, but it enters into my milieu through complexity science. I also find that physics offers a useful antidote to the pseudo-certainties of economics, as I’ll discuss below.
Time’s arrow
For a novice like me, Rovelli’s The Order of Time is a particularly good account of how something which we take for granted as a straightforward reality in our everyday experience — time — is often very different from what it seems. Time’s arrow seems to flow from the past to the future, and appears to be constant, but it actually passes faster in some places (mountains) than others (sea level).
According to Rovelli, time’s arrow is only real in relation the second principle of thermodynamics:
‘Heat passes only from hot bodies to cold, never the other way around… This is the only basic law of physics that distinguishes the past from the future…. The arrow of time appears only where there is heat… every time a difference is manifested between the past and the future, heat is involved.’
But then, we perceive time to be real as past, present, and future in some way (at least in the Western world). It is real from a particular point of view, rather than as a universal truth. As Rovelli puts it:
‘The history of the happening of the world itself can only be an effect of perspective: like the turning of the heavens; an effect of our peculiar point of view in the world. . . . Inexorably, then, the study of time does nothing but return us to ourselves.’
So, if something so fundamental as time is really just a matter or perspective, why do we hold so dearly to a supposed objectivity in social science? A key reason for this is that the disciplines we are trained in (e.g., economics, sociology, physics) sediment not just a set of agreed “facts” but also concretise an epistemic viewpoint— that is, what and how we can know things about the world (and universe) around us.
At the Danish Evaluation Society Conference in 2023, Anna Folke Larsen told me a joke about economists that really rang true, It’s told here:
A physicist, an engineer and an economist are stranded in the desert. They are hungry. Suddenly, they find a can of corn. They want to open it, but how?
The physicist says: “Let’s start a fire and place the can inside the flames. It will explode and then we will all be able to eat”.
“Are you crazy?” says the engineer. “All the corn will burn and scatter, and we’ll have nothing. We should use a metal wire, attach it to a base, push it and crack the can open.”
“Both of you are wrong!” states the economist. “Where the hell do we find a metal wire in the desert?! The solution is simple: ASSUME we have a can opener.”
Anna is an economist by training, so I feel she has the right to make such a joke. Perhaps I don’t. But, economics is unfortunately the discipline which most often imposes itself on all the rest of us, rather than promoting transdisciplinarity. I know several very smart economists, but they are smart (in my view) because they read widely beyond their field and are usually very critical of their field.
Situated perspective, confidence and consistency
Rovelli’s view of perspective also matters in how we understand fields of study like complexity science and the debates within fields. Dave Snowden wrote a particularly elucidating blog comparing Cynefin and the Stacy Matrix. In the blog, Snowden questions whether we have been separated by a common language. I’ve read a lot of both Snowden and Stacy, and Snowden’s blog really helped me to understand why he holds such a radical position on knowledge and knowability.
Stacy, a management professor originally trained as an economist, argued that natural science could not be applied to human systems except as an analogy. But, Snowden notes that:
“Stacy… takes a phenomenological stance (the axes labels alone [in his famous matrix] create this) while Cynefin is an ontological framework that allows radically different conversations mediated by epistemology and phenomenology. To make that simpler: how things are, how we know things and how we perceive things are distinct and increasing the alignment between the three is key to coherent action.”
For Snowden, complex systems in nature actually behave in a particular way, and thus we should not view complexity as mere analogy. I’ve always been more sympathetic to Stacy’s position due to my background in the humanities. To me, natural sciences apply quite awkwardly to social sciences, and are thus a better fit as analogy. I’m thus quite comfortable with Rovelli’s commentary on the scientific nature of time, but also with his point on the importance of situated perspective.
Andi Fugard, a thoughtful cognitive scientist and evaluator, once told me they found discussions about epistemology unenlightening. Fugard is unusually sensitive to the importance of positionality, but I have found that evaluators who chiefly focus on quantitative methods (like Fugard) tend to be less comfortable with epistemic relativism. Often, (though not for Fugard who, unusually, rejects positivism) this is due to a positivist stance. Words like “causal” and “objective” are, more often than not, employed as absolute properties that a research or evaluation design either possesses or not rather than a series of greys mediated by degrees of knowability and bias. These words tend to express a sense of certainty, when what we actually need is greater comfort with uncertainty.
Indeed, when there is a sense of degrees, there is usually an arbitrary threshold (e.g., p < 0.05) which radically transforms our confidence in findings. As several editors from American Statistical Association suggest, this is dubious. Statistical significance and your p-value are not as important as we think. There are not single magical numbers (p = 0.051 vs. p = 0.049). Instead, they argue that ‘researchers [should] be free to communicate all their findings in all their glorious uncertainty.’ And this does not merely apply to statistics, it also applies to set-theoretic methods which have a statistical flavour. For instance, I recall discussing with Rick Davies about consistency thresholds for Qualitative Comparative Analysis (QCA). It’s typically recommended to have consistency levels of 0.75 for clean set QCA or 0.8 for fuzzy-set QCA. These sound like sensible levels, but neither Rick nor I can find a sound justification for why these specific numbers delineate a clear line between a pass and a fail, or inclusion and exclusion.
Knowledge is situated, positional, and plural. Who we are and where we come from shapes what we believe and why we believe it. I’m an identical twin. And this has doubtless shaped my lived discomfort with experiments and the assumptions we have about selecting control groups. Even minor environmental differences can make a significant difference in trajectories. We process the same information or respond to the same intervention quite differently, despite significant biological similarities. I have a philosophy undergraduate degree and I was always struck by Jean Paul Sartre’s view of double consciousness. As I recall, he describes the dissonant experience of seeing oneself and how others see you as being different, and yet both phenomena are real. This has always shaped why I’ve been uncomfortable with positivism. Sartre’s famous dictum of “hell is other people” from his play No Exit is about how we can’t control how others see us (we are trapped by how others judge us), and this generates a sense of anxiety.
A doubting Thomas: Better data ≠ better decisions
So much of what we claim to know in the world are incomplete pictures, unfinished stories, and partial truths. There is so much in this world that we simply don’t know, and we should be more comfortable with doubt. And yet, the “evidence-based” policy revolution so confidently tells us “what works” and apparently what doesn’t, usually based on a small number of studies from an extremely narrow range of evidence and acceptable designs, usually Randomised Control Trials (RCT).
In Randomised Trials as a Dead-end for African Development, Seán M. Muller argues that one of the main problems with the randomista project is that it ‘smuggles in ideological and epistemic bias,’ with ‘missionary zeal’ and ‘incredible certitude’ about their findings. I think the missionary analogy is apt because so much of this world of zealous certitude seems to be an unwise, costly, and fragile crusade.
This came up in my news feed recently when reading Tim Harford’s ‘Known unknowns’, or how to plug the gaps in public research where he made the case for funding more systematic reviews, and the International Initiative for Impact Evaluation’s (3ie) “widely admired… Development Evidence Portal.” As I have discussed previously, the Portal sounds like a great idea, but it has many problems, particularly for areas of policy that are more contested and where the evidence is more mixed, because context has an important bearing on results.
Harford argues that there ‘ is no surer way to identify gaps in research than to put together a systematic review.’ What his conclusion is based on is unclear. There are numerous ways to assess research gaps (literature reviews, umbrella reviews, realist reviews, etc.), and it’s uncertain why a systematic review is necessarily the best way to plug gaps (gap plugging also seems to be an unhelpful analogy). Particularly when reviews are based on a very narrow range of evidence (as is common practice), they are more likely to provide a misleading picture of both the “known” and “unknown,” and this, in turn, will likely lead to misleading policy recommendations.
There is also a fallacy here in the belief that more “better data” (i.e., RCTs) is necessarily the answer. As Jen Riley put it recently, more data is not the answer to knowing what works. Nor is the solution merely the better organisation of these “better data.” It’s not simply a matter of plumbing, plugging research gaps, or even producing more (supposedly) “rigorous” studies. Riley argues that “better data” (by which she means RCTs) is a red herring, and I think she’s largely right. For Riley,
“Ultimately, the call for “better data” should be reframed as a call for “better culture” in which we foster a culture of learning, adaptation, and honest reflection.”
In my view, key elements of this culture include:
- Comfort with degrees of uncertainty about “the evidence” (plural);
- Centrality of context (and relevance) in the interpretation of evidence;
- Open discussion of “the evidence” with multiple stakeholders and viewpoints, and;
- Reasoned and transparent evaluative judgement, taking each of these elements into account.
- Reflecting on the “so what” and “now what,” not just the “what” works.
What we need, in short, is a situated debate about “the evidence” to make better decisions in the real world.
