Day 3 take aways: Are we there yet?
This blog is one of a daily series crafted by James Oriel to capture his insights and reflections from the day. They can only hope to provide a quick peek into the rich discussions held, and we hope they offer an invitation in for those who may wish to explore further via the recordings, links and other resources provided.
Half the challenge of creating value is learning how to recognise it
Targets can be useful, guiding our actions, determining our incentives and forcing clarity around what our goals actually are. Trouble is they’re easy to game, rife with power dynamics, and prone to all manner of distorted outcomes. No wonder then so many of us in this work can feel like we hit our targets but completely miss the point. Yesterday, we spent the day exploring practical approaches to making sense of change in complex, evolving systems and the critical role that evaluation plays in guiding learning and innovation. Enjoy the festival recordings on our resources page to catch up on at your own speed, and read on below for a recap of the day.
Watch the recordings on the festival resources and recordings page.
A few things we learned yesterday
A lot of our first session was about taking a big step back from having any preconceived notion of what success looks like. This doesn’t come naturally to many evaluators, typically trained to and tasked with creating frameworks to make sense of whether something is working, or valuable… or not. In this dynamic the often unspoken rule is that the project commissioner, their most powerful stakeholders, or even the evaluator themselves heavily influence what constitutes success. But what if that were different? How would you begin?
Well, we began in Australia, with resident Aunty and Aboriginal lead at TACSI, Aunty Vicky Charles. She reminded us that everything starts with relationship building, and of the simple, overlooked power of having a good yarn, how chatting and really listening can reveal so much about what people want, worry and care about.
This reciprocal way of being echoed throughout our first session. Angie Tangaere described how her work at the Southern Initiative, rooted in Indigenous knowledge systems, has helped her to “move away from planning, programming, service delivery, towards ‘attuning’: attuning to the planet, attuning to each other, attuning to rhythm, attuning to what space looks like and attuning to relationships as a fundamentally different way of being ”, in order to reverse-engineer the systems that are holding inequity in place. That comes about by steeping ourselves in context, to allow a sort of “slow cooked” approach to evaluation, as Jess Dart described it, to emerge from the real needs of those most impacted. This required plenty of unlearning: moving away from being accountable to pre-determined targets, to being accountable to “learning our way through change”, she says, and becoming more of a ‘partner’ in uncovering the patterns that point us towards a better understanding of what systemic shifts look like in practice.
And to close, as a funder Niall Fay is repeatedly told that their work — funding place-based change in mental wellbeing through ten-year, trust-based grants — is brave and bold. But really, he argues, it’s good sense. Because if our old approaches – to evaluation, to funding, to innovation – haven’t worked, trying something radically different is really your only move. And maybe that radical move is as simple as “sitting with communities in genuine partnership”, and a large part of that, is collectively deciding what good looks like - where the community themselves identify the indicators and markers of change they want to see.
Charlie opened our second session by outlining that shifting systems is a collaborative, uneven, protracted and evolving process. Understanding where you are, and who you are, in that process is a challenge in itself.
Too often, Ben Ramalingam says, evaluation is associated with internal bureaucratic failures. It ends up becoming the only mechanism in an organisation for telling senior leadership all “the bad news about their flawed assumptions”. Which goes down as well as you might expect. Far better to test your assumptions early, and often, than to take your medicine all in one go when it’s too late and you’ve spent all your budget.
This point touches on an overriding theme that the assumptions embedded into what is evaluated (and when, how and by who) are too rarely made explicit. They’re not made open to challenge and rarely updated quickly when proved wrong. The faster we can burn through all the bad assumptions the faster we get to an idea shared sense of what good looks like. This reflects an accountability to learning that Lee Alexander Risby spoke to, and echoes Jess Dart’s comments in our previous session.
That gets harder the more zoomed out from an issue you are. Anne Bergvith Sørensen talked about the need for different approaches to evaluation at the different levels of the system that the Homes For All Alliance works at to address homelessness. Programme evaluation can work for innovations at the micro level, but at the macro level, where the Alliance is working to change the discourse around homelessness, it is a case of “trying to track changes in what is said and what is written about homelessness and create a new story, a new landscape”, asking stakeholders what has changed and what have the Alliance members meant to that change, and acknowledging that they are only one part of a broader movement that is shifting the system.
At the heart of our programmes there should be an evidence-based perspective — “an engine for intuition and action”, say Ben Ramalingam. But the danger is that calling it evaluation will quickly link it to a bureaucratic function: judge, jury, and executioner, rather than guide, Sherpa, navigator. Making sure we don’t undermine the important role of evaluative thinking in helping us navigate complex situations requires a different bureaucratic intent.
So how do we bring this evidence-based intent to our work, without crushing our intuition and ability to evolve together in complex environments? Because we do also need evidence – for ourselves and the many we need to convince to join our efforts along the way. Chris Clements helped frame this challenge, reminding us that, of course we want to understand if we are getting closer to the change we seek, “it’s why we are here”, but the question becomes how to do this without overclaiming impact, distorting outcomes, or glossing over the real complexity that exists.
In our third session, Bonnie Chiu shared how, for funders, the shorthand ‘ToC’ stands for theory of change, but for many grantees, it simply means Tool of Control. What is otherwise a useful way to draw out the assumptions made for how a programme might work, is too often experienced instead as a way of ‘holding accountable’ those funded to do the work, with the challenging power dynamics of that, especially in places that have formerly been colonised. And yet, Bonnie says, evaluation can itself be an act to create equity and inclusion — not just passively studying or evaluating something, but playing a role in enabling communities to generate knowledge: “after all knowledge is power. And evaluation helps create knowledge. It extracts knowledge, but it can create knowledge if it stays within the communities. It actually is a tool for Equity and Inclusion.”
In a similar vein, Anna Folke Larsen shared a new way to balance the often rigid requirements that ensure the rigour in impact evaluations such as random controlled trials, with the need to empower front-line staff to apply their own judgement and creativity when faced with complex, emergent situations. Defining the intervention through a set of principles to be enacted, rather than a strict protocol to be followed to the letter, makes it possible for staff to use their own creativity whilst maintaining a strong enough basis for comparison between groups. This in turn requires plenty of creativity on the part of evaluators themselves in designing the evaluation, but as Anna says, impact evaluations of principle-based interventions can also shed a broader light on what works than the narrow beam of a traditional RCT.
That can be at least a stepping stone in understanding the process of shifting a system that may take many years. “You're not really running an impact evaluation of an entire system shift, because it's very difficult to see what would have happened, had we not done these efforts to change the system. So the idea is actually a balancing act of having principles that hold the promise of shifting the system. But trying to do it in a way where you can still hold on to some comparison groups. This is a way that you can try to assess the stepping stones of the change and so the system change potential that these principles could hold.”
It’s one thing for evaluators to call for a different approach to evaluating complexity; but it is another for large organisations to listen, says Nina Strandberg. “…the evaluators have been telling us this for a decade, and we hadn't listened or we didn't have the possibility to act on what they recommended. It was really good, but we didn't act on it. So what we did in essence was to build a movement.” That movement, built of ‘positive deviants’ across the organisation starting to act in new ways, helped SIDA, then an organisation with a strong culture of risk aversion, to move away from treating complex challenges as technical problems to be tightly managed and controlled. Removing the obligatory ‘results matrix’ was an example of how the accumulation of small can be big: many acts like that changed the cultural focus on risk and control and shifted SIDA’s internal system.
The Omidyar Network felt that too, and shifted its evaluation approach away from a narrow sense of impact and accountability after the realisation that “whether we get impact for our dollars is, like, 95% out of our hands” said Rob Ricigliano. And so learning, rather than impact, became its top priority, marked in shift from holding regular ‘results and accountability’ sessions to holding ‘learning and stewardship’ sessions. Across the Network each team now has an embedded learning officer so that learning cycles is speeded up. What was evaluation for? To help them be more effective in their work. That meant evaluation as a near constant learning system.
Rob’s work also keeps track of three different levels of a system: supportive actions, that increase the probability of certain conducive conditions, that in turn support a shift in the system’s health. The conversations you have and the tools you use are quite different at these different levels: “You have much more agency and even the ability to attribute cause and effect relationship [at the level of supportive actions], than you do at the conducive condition level because there's a lot more forces that come in to effect those conducive conditions… At the systems health level, you do need to ask, what do we know about whether this system is healthy or not? That's divorced from whether we're impacting it, because you can't make that jump. You can't go from action to systems health directly.”
This is also what Yaera Chung’s work on economic growth at UNDP sought to explore with municipalities in Armenia, Azerbaijan, Georgia, Moldova, and Ukraine. Yes, include some measures for particular outcomes, but also how can we understand the dynamics of the system that produced those outcomes? What is the capacity of a system, the culture of it? Yaera has created complex system portfolios with each municipality team, and the monitoring of those portfolios is designed to help teams learn about options for positive system change — but also to increase their own capacity to learn and adapt. For her colleague Søren Vester Haldrup, it’s not really about any one right way of doing it, but rather being aware of and intentional about what approach you take and why. Which means, and evaluators will probably sigh, involving evaluators from the beginning of projects to design those approaches, refining the principles that they will be based on. It’s rare that evaluations can be stuck on at the end with any success beyond ticking the ‘we did an evaluation’ box.
For Beth Smith, it all comes down to who does the interpreting, reminding us that whichever interpretation prevails is a function of power, rather than truth. Beth’s work at The Cynefin Company uses storytelling to surface hunches and underlying beliefs, and to capture the broad diversity of experiences and perspectives present in a system at any one time. To keep those perspectives in play throughout, Beth holds on to three principles: reduce the granularity at which you measure a system (“so talk about the small stuff, do it en masse”); disintermediate (“remove the middle person's interpretation”); and distribute cognition (“so make the sense-making of those instances a social process”). Aggregating those stories reveals common patterns as well as differences, meaning that Beth can build up a picture of change at different levels — individual stories, community stories and the national picture — and where individuals fit into that bigger story of change: “If you do that over a period of time, we're able to use this concept of what we call a vector and start to map the speed and direction of travel, rather than pre-defining an outcome.”
And Beth summed up that feeling that our entrepreneurs on Day 2 also spoke to, of being caught between two systems: “… it’s what I would call domain dissonance — it’s like you could be living in one type of system, but applying the rules and logics of a different one. So for example, you know, we're living in this really complex environment with all of these unknown variables, yet we're still measuring them as though we still manage and know all of the variables”.
Our closing session of the day was titled ‘how to create value’ but could also have been called how to recognise it. Evaluation is a form of testing reality, it’s the way that people — all of us — come to understand, or make sense of what’s important, what has value. Because you can’t escape the fact that evaluation is about valuing things. And that is, as Mark Cabaj noted, about our “diversity of experiences”. It’s about what is valued, and by who. And that’s not neat and tidy, because neither are people. We respond to different things. So it’s messy. And that means evaluators have to create the space for dialogue in which an intervention strategy can be understood from many people’s perspectives “because if we could do that, and you could do it reciprocally, we might have the possibility of doing something generative — to be more than who we are, to synthesise - and not just intellectually but emotionally too”.
But that’s not really how a lot of evaluative work is carried out. Indy Johar framed the flaws of evaluation, as commonly practised, in terms of “a theory of power”, because it’s so often about who decides what good looks like, and imposing those judgements on others. So it’s about power, and about control, something rooted in “the tyranny of management”. Instead, he suggested the more useful framing is to think of evaluation as learning to understand what the capacities and capabilities of a system are; what can it do and how does it do it?
Which sounds slightly different to the idea of having milestones and a guiding north star, or mission, or direction, something brought up again and again across the day. Jessica Davies told us how “inherent in systems change work is the ambition that we're in this change to have impact not just to kind of redesign systems for the fun of it”. She pointed to teams hired in as ‘learning partners’ but kept entirely separate from the official evaluation teams, compartmentalising something that should run hand-in-hand.
For evaluation to be generative — for it to play a true role in opening up the potential for better future systems — means evaluators need to “put away our old tools, our old sets of assumptions”, says Emily Gates. That’s because evaluation tends to look back at the ‘thing’ we did, with its boundaries more or less set. But if we are looking forward, to understand the multiple scenarios that might be playing out, “those boundaries are pretty different. We are constantly changing what scenarios we’re looking at. And that means both what facts become relevant and what value bases become irrelevant”. That is a different undertaking to what evaluators have been trained to do: it becomes a much more collective, deliberative process of co-creating value, rather than judging worth.
Outcomes matter, as long as they are being generated through the right kinds of relationships and dynamics — and they are outcomes that we care about in the long term. This is where collaborating with a wide range of people to work out what that good looks like, to get a sense of are we there yet (and where is there) can be so powerful. Whatever the work is, whether small project or major programme, we also must remember it sits “inside of something bigger, it's inside of something much more ongoing, and it serves that longer term purpose”, says Emily. Evaluation that takes the long view, and realises our contribution is inevitably just a part of a larger and quite unpredictable pattern.
Charlie ended the day with a story, of a dispute between Rosa Luxemburg and Lenin over the nature of the Russian Revolution:
“Lenin said ‘Nothing's going to happen here unless there's leadership to give direction’. And Luxemburg said ‘No, no. Actually what we have to do as revolutionaries is read the signs of what the aspirations of the masses are, and then help give form to them”. And so Lenin said leadership is direction, and by implication, control; and Luxemburg said no, leadership is learning.
Two quotes
“the best thing we can do is to share the power and control that we have, in a way that empowers people or enables people in place to design what the future looks like… [it’s] localised, strength based, and Indigenous led, building on what works for people.” – Angie Tangaere
“All things are subject to interpretation. Whichever interpretation prevails at any given time is a function of power, and not necessarily truth.” – Nietzsche, as quoted by Beth Smith
A bit of housekeeping
We’ve also got an informal group chat going over on LinkedIn, it’s free to join, and a place to continue the discussion over the whole week. Join the festival group on LinkedIn.
As well as the resources our speakers share, full quotes, images and recordings are online for you to explore the whole festival programme at your own pace. Check out the Festival recordings and resources page.