Day 3 ⏐ Session 3

Written By Johannes Björkman

Are we making progress?
New evaluation approaches for system innovation in action

New approaches to evaluation not only support system innovation efforts, but can be themselves system shifting. We explored how even in embracing the language of ‘learning and stewardship’ and moving away from terms like ‘results and accountability’ indicate deeper attitudinal shifts.

Our panelists in this session take this further still by erring away from unhelpful extractive evaluation processes in favour of a kind of knowledge creation that can be a force for inclusion and equity where people are sharing their stories and making sense of them for the direct benefit of their communities and the things that matter to them.

Highlighting new practices in action we held this conversation with; Nina Strandberg, Swedish International Development Cooperation Agency (SIDA) (SE); Søren Vester Haldrup and Yaera Chung (US and Central Asia) on the UNDP’s Monitoring and Evaluation Sandbox; Rob Ricigliano, Omidyar Network (US); Bonnie Chiu, The Social Investment Consultancy (UK); Anna Folke Larsen, ROCKWOOL Foundation Intervention Unit (DK) on impact evaluations of principle-based interventions and Beth Smith, Cynefin Group (DK) on the role of mass storytelling in uncovering the trajectories of system change.

“I think in the question of ‘whose voices matter?’, ultimately, I think it’s for those with power to recognise that their voices automatically end up mattering a bit more. How do you have the humility to take a step back and to perhaps ensure that at the end it is the communities that you’re seeking to serve, to put their voices are at the forefront of driving the change that they want to see, and that the outcomes and the change are defined by them rather than by external evaluators”
— Bonnie Chiu

Resources

Quotes from the session

  • The shift at Omidyar… “was they changed from an annual result accountability session to a learning and stewardship session. And largely that was because the deeper macro level realisation was that the driver of your success is much more linked to learning than it is to near term, proximate, measurable, attributable impacts. And so that was the big shift and that led to the idea of stewardship. Whether we get impact for our dollars is like 95% out of our hands.” (Rob Ricigliano)

  • “Shifting to those two words, to ‘learning and stewardship’ was really reflective of a much deeper attitudinal shift at multiple levels” (Rob Ricigliano)

  • At Omidyar “each programme team has an embedded learning officer or a person whose job it is to shepherd that team's learning processes. So that's a huge shift, right? So that learning happens much more on a quick cycle time. It's not an annual thing and it's not for the purposes of demonstrating impact. It's actually for the purposes of increasing your effectiveness in the work that you're doing” (Rob Ricigliano)

  • “...there's already an undercurrent… which is the power dynamics between funders, and the people they fund. And as part of that dynamics, you can also think about the racial dynamic as a lot of funders are located in global North… so that's really the context within which we are we have to think about it, to interrogate where evaluation then perpetuates those power inequalities” (Bonnie Chiu)

  • “...accountability is seen as controlling and we're not bringing the best out of people and when you layer those racial dynamics, colonialism, history of colonialism, it really gets very complicated.” (Bonnie Chiu)

  • “...knowledge is power and evaluation helps create knowledge. It extracts knowledge but it can create knowledge if it stays within the communities. It’s actually is a tool for equity and inclusion.” (Bonnie Chiu)

  • “...before we secure projects with clients, we [ask] how comfortable they feel about embracing failures? We find that if they are truthful in answering that question, or if they're not, it really tells us to what extent they have the appetite for change.” (Bonnie Chiu) 

  • “All things are subject to interpretation. Whichever interpretation prevails at any given time is a function of power and not necessarily truth. And so for us that is kind of the crux of what we're interested in… we can harvest all of this information through stories, statistics, multimedia, but the power in doing something with that is the power in interpreting what does it mean and what should happen as a result.” (Beth Smith)

  • “So what we're saying is that the person who shares a story should also be the person that analyses that story because they know it most intimately and it's subject to their cultural bias which is appropriate for them. And our system works then to aggregate everybody's stories together, so that we can start to see culturally where there might be a kind of common belief, common experience or actually even difference within the same system.” (Beth Smith)

  • “Why is it that old wives tales are called old wives tales when old men's tales are called Philosophy and Religion” (Beth Smith)

  • “I think there's a really important distinction to make particularly within a complex system. A complex system is not the sum of its parts but the product of their interaction.” (Beth Smith)

Johannes Björkman
Previous

Day 3 ⏐ Session 2
Next

Day 3 ⏐ Session 4