Day 3 ⏐ Session 4
Helping to create value:
new roles for evaluation and evaluators
Evaluation is a form of testing reality, it’s the way that people – all of us – come to understand, or make sense of what’s important, what has value. You can’t escape the fact that evaluation is about valuing things. But here rests the continuing debate from the first few sessions… what’s being valued and by who?
This brings us back to the problem of power control, something rooted in “the tyranny of management” according to Indy Johar. Instead, he suggested the more useful framing is to think of evaluation as learning to understand what the capacities and capabilities of a system are; what can it do and how does it do it?
This conversation was held with leading practitioners on what evaluation could become in the setting of system innovation: Mark Cabaj, From Here to There and Tamarack Institute (CA); Emily Gates, Lynch School of Education & Human Development, Boston College (US); Indy Johar, Dark Matter Labs (UK) and Jessica Davies, Social Finance (UK).
“It’s not that the evaluator says this is the best path, or this is the path that has the most votes. The idea in my view is that we say let’s carefully understand the consequences of each of these paths as best we can and then deliberate about where we should go… there’s an ethical justification for the path chosen.”
Resources
Quotes from the session
“Evaluation as is a very natural act and it’s probably better framed as reality testing. How do we know what we think we know? Is there another way of understanding this? What are the other perspectives on whether this is good, bad or uncertain?” (Mark Cabaj)
“Evaluation could also contribute to the act of transformation by saying, do we after an evaluative exercise have greater insight and empathy into someone else's perspective in a pluralistic world, greater civic literacy about the systems that we're trying to change and what holds them in place and some opening set of ideas about how we can do something better than what we were trying to do before.” (Mark Cabaj)
“Evaluation is not tracking measurables, that's part of it… Evaluation is the act of valuing.” (Mark Cabaj)
“I really think that evaluation needs to sort of come up to speed… some of that means putting away our old tools, putting away our old sets of assumptions and seeing what's been created in the spaces where people are really doing it in that kind of way in that co-creation value kind of way.” (Emily Gates)
“Our expertise is based in our Social Science Methodology more than anything, and maybe our social science theories as well. But we're less equipped to say what did we do when there are really different values? And when those really different values are based in different worldviews? How do we facilitate deliberation that's generative?” (Emily Gates)
Rethink Health “use this concept of system stewardship… Outcomes matter as long as they're being generated through the right kinds of relationships and dynamics. By that, I mean, they're set up to be designed for the outcomes we care about over the long term rather than producing outcomes in the short term we think we want and then go away.” (Emily Gates)
“I think the evaluation word is problematic because I think it implies a theory of power… who's evaluating to what ends and there's a really interesting question about that. That opens up, I think, a better language, which is, in my view around learning.” (Indy Johar)
“I think system capabilities are much more critical in system outcomes or outputs. I think the system capabilities and the self adjustment, the capacity and the agency to self correct, the responsibility and to be accountable to self correct, has to be at the agent level of the system, the actor level as opposed to at the meta evaluation level.” (Indy Johar)
“Why is pluralism important? It's not pluralism, it's democracy or the distribution of agency. And I think we have to talk about the tyranny of management. Management comes from kingship, management comes from master and slave theories of employment. There's a logic of control, which has been broken and I think this is kind of the next emancipation of democracy” (Indy Johar)
“Change makers make choices by drawing on multiple sources of intelligence. Some of it is evaluative feedback, formal, some of it is just what's going on in the field, their own intuition, etc. And so I think that more accurately captures how people make decisions” (Mark Cabaj)
“...most niche or innovative transformative projects have almost no chance of scaling, because they're so far ahead of the ecosystem that you know that there's too many things that hold them back. So their purpose isn't, did you achieve an impact and should you scale? It’s, have you demonstrated an alternative? Have you disrupted narratives? Do you give people hope that there's something else? Can we fill up the pipeline so if the manifest failure of the current system happens we have things in motion ready to go? And that’s a long term game…” (Mark Cabaj)
“To me the main thing is… team science meets citizen science, if you will. So it's like those have to come together and disciplinary and methodological pluralism are really important. Plus the actual like experiences not an over crediting of certain kinds of expertise to go back to that distributed idea.” (Emily Gates)
“It's the how of putting that team together and the principles of putting our team together more than having a model of the who so that we don't end up with the kind of blueprint problems we've had in the past but actually just think a bit more deeply about how we're assembling those things in different contexts.” (Jessica Davies)
“I think the biggest thing would be actually someone to deal with historic trauma. I think most people that are operating in organisations and leadership places, the higher up you go, have had systemically been made violent by their journey there. I think unless you can unwind that it's so difficult to deal with systems” (Indy Johar)
“I’m reminded of the dispute between Rosa Luxemburg and Lenin over the nature of the Russian Revolution… Lenin said nothing's going to happen here unless there's leadership to give direction. And Luxembourg said no, actually what we have to do as revolutionaries is learn what people want. We have to read the signs of what the aspirations of the masses are and then help give form to them… And there's something really fundamental about that, which is leadership is learning, learning interchange and allowing a sort of communal collective learning, that's been at the heart of that's been at the heart of what we've been talking about.” (Charlie Leadbeater)
