Research meets evaluation; evaluation meets research
Are research and evaluation really that different? And how much does it really matter?
Rockford Sandsom argues that ‘in an everyday context, research means finding information or compiling and comparing information.’ By comparison, Maria Bustelo defines evaluation as the ‘systematic collection and analysis of information (my translation).’ These sound very similar to me — both are about finding, collecting, and compiling information, and analysing it. But, surely there are some important differences?
Evaluation used to be considered a form of applied research (see Weiss, 1972). And yet, for Michael Scriven, evaluation is a transdiscipline — ‘an essential element in all other academic disciplines (see Honoring the Legacy of Michael Scriven for a discussion).’ Andrew Hawkins goes even further, arguing that ‘evaluation is life.’ For Hawkins:
‘To evaluate is to survive. It may be pre-cognitive, but fight, flight or freeze; explore, exploit, or conserve are driving forces of survival.’
There is something elementary in our prosaic understanding of evaluation. It’s just a thing that humans do to sense, interpret, and understand the world around them. But, is this prosaic understanding helpful? I’m not so sure it is. Almost all forms of human experience would be encompassed under such a broad definition.
In What Is Evaluation?: Perspectives of How Evaluation Differs (or Not) From Research, Dana Linnell Wanzer included 12 canonical definitions of evaluation. As Linnell points out, most definitions are based on Michael Scriven’s definition which emphasises the ‘merit, worth, value, or significance’ of something. Fundamentally, evaluation is about values and valuing, as I’ve discussed previously. Linnell’s survey of American Evaluation Association (AEA) and American Educational Research Association (AERA) members found that ‘most participants believed the purpose of evaluation is to provide a value judgment (84.4%).’ Another definition I like comes from Carol Weiss (1997: 4) who defined evaluation as:
‘The systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy.’
The reason I like this one is because of Weiss’ emphasis on a yardstick (i.e., standards).
Another key element is judgement. Patricia Rogers also has a nice summary which links standards and judgements:
‘Evaluation is a judgmental process, involving the assessment of findings/observations against standards, for the purpose of making decisions, asking such questions as “What is/was good?” or “Which is/was the better?” or “What conditions are the best to nurture to produce desired results?’
Julian King later added evaluative reasoning into the mix, emphasising: criteria, standards, and evidence as part of the elements through which evaluators make evaluative judgements about the value, merit, and worth of something.
The comparisons evaluators make with research, however, I tend to find unsatisfactory, as they often tend to be a straw man critique of the worst Ivory-tower you can possibly imagine. Even what appear to be good faith assessments seem to be written by evaluation scholars critiquing non-evaluation scholars in absentia. I don’t mean to name and shame contributors, but I would seriously encourage evaluation scholars to consider action research and the many attempts at maximising research impact that many researchers pursue. Many researchers do a lot of things evaluators claim to be unique to evaluation. Why evaluation must be unique I have never fully understood.
Nonetheless, I’ve had a go at making a few potential distinctions. I often find that if you’re not sure what a concept really means or what its boundaries should be, you should look at canonical definitions (such as those listed by Linnell) and then go back to word origins — etymology. If we do this, potential distinctions often become clearer. Below you can find my brief summary:
Of course, all of this is debatable.
If we take the etymology seriously, values and valuing are the centrepiece of evaluation. Many evaluation scholars have discussed this previously. However, this doesn’t mean that researchers don’t have values or that they are somehow value neutral, as some evaluators assert. The very claim of value neutrality, by evaluators or researchers, espouses a particular kind of value position. One of my favourite diagrams from Brown and Dueñas on research paradigms clearly shows how what we value shapes our lines of inquiry in research as well as evaluation.
For Andrew Hawkins, evaluation is also about the study of rational and ethical action (praxeology). Perhaps it should be, but there are abundant examples in evaluation that reflect very little (if at all) on the ethics of action (many field experiments, in particular). I was amused to find that the clearest reference I could find to praxeology and evaluation was actually in the SAGE Encyclopaedia of Action Research. That pretty much makes my point for me on the artificiality of clear and clean divisions. I mean to write a future blog on why ethics ought to be the alpha criterion in evaluation, but perhaps it should be in research too.
In terms of some productive overlaps, often dubiously portrayed as distinctions, I would argue that both evaluation and research:
- Are systematic assessments;
- Often emphasise the importance of use;
- Are often geared towards decision-making and improving public policy;
- Can be practical or impractical.
If at all possible, we should propose an a fortiori view of those whose views (or in this case field) we mean to differentiate ourselves from, representing strengths at least as much as weaknesses (I am sometimes, but not always, guilty of failing this when talking about randomistas).
Evaluation doesn’t have a monopoly on utility, decision-making, or practicality, whether we view evaluation as the “alpha discipline” or a sub-discipline of research, or even science. Linnell offers some useful illustration of some possible relationships between research and evaluation which can help us think through the potential overlaps:
Patricia Rogers also offers some trade-offs on different ways of framing the differences. I think she’s right that there some benefit to portraying the connections in different ways.
For me though, the non-hierarchical and complementary option D is the closest to my understanding and experience. Linnell argues that this is ‘the most popular viewpoint, particularly among evaluators.’
Linnell’s study suggested, however, that ‘evaluators are more likely to see the differences between evaluation and research compared to researchers.’ As mentioned above, I think this is because the field of evaluation has expended considerable efforts in attempting to break away from its older sibling of research, whereas the older sibling has expended very little energy in considering the supposed specialness of their younger sibling.
There are, often, differences between evaluation and research in value-orientation, focus of inquiry, types of questions, and primary units of analysis, but few if any of these differences are necessary. There are so many areas where they two disciplines/fields of inquiry overlap. So, in my view, we should spend less time in explaining how we’re different and more time searching for synergies to move both fields forwards in a common direction which emphasises common values and uses — a future which is ethical, inclusive, and reflexive.
