The following is an extract from the two meeting on Measuring What Matters, published here and here, and repurposed into a written Q&A to extend this conversation to a wider audience.
On 9th May 2023, the Global Fund for Community Foundation hosted a meeting with Deepthy Menon in conversation with Barry Knight on Measuring What Matters. The meeting was premised on the context that communities most proximate to the development challenges they face often invest their own resources in addressing their local development issues. However, this local investment by communities is rarely acknowledged by the international development system. Out of this, an inclination to mainly measure quantitative success as a process of vertical accountability has been adopted at the expense of horizontal accountability to the communities co-investing in solving their development challenges who may choose to measure qualitative elements such as trust and dignity. The meeting was a continuation of the work of over 130 people from civil society around the world who contributed to the consultation paper on Measuring What Matters by Barry Knight and Dona Doan who concluded that organizations need flexibility in what and how they choose to measure. The following are extracts from the meeting repurposed into a written Q&A to extend this conversation to a wider audience.
Barry: The first thing to say about the report on Measuring What Matters, is that it lists 130 people who contributed ideas to the report. So, it was crowdsourced in a very important way, and as part of the work, we went to several places. We began in South Africa. We went to Serbia, Brazil, Russia, Canada, and the United States. We also held monthly zoom calls too. The report grew out of activists from all over the world, North and South. So, although we can formally say it was written by two people, it was much wider than that. This is the kind of philosophy that we want to continue because, as SBAM said, the way the monitoring evaluation field sets up very expensive experts as consultants to look at the field often denies the fact that there are skills, knowledge, experience, and expertise within those communities.
Much of the work that the global science constituency does is working and developing where there is nothing. The key thing is to develop stuff. The word that you’re going to hear a lot of today when we get to talking with Deepthy, is the word – emergence. Things that change systems are things that emerge from the ground up. That is really where measuring what matters differs from the standard monitoring evaluation text. We are going to be talking with Deepthy Menon from the Urgent Action Fund. She will be setting up a well thought and constructed paradigm of doing things differently so we can learn from people’s experience and lift different ways of doing things. So, I’d like to, if I could, to invite her to us a bit more.
Deepthy: I’m a storyteller who got roped into the role of being a monitoring and evaluation person. All I had with me was the audacity to say it’s not difficult to figure out what’s required. I remember the pitch I made when I was being interviewed for the job. They said, you don’t have traditional monitor and evaluation experience. I said, “but I worked in a non-profit which had three impact indicators for impact evaluation.” It was one of those big impact evaluation funders and I was working with them as a senior communication manager, so I knew what it entailed, but I also knew what might not be required in all contexts. So, there was also data driven decision making, which is seen as very important. All big, middle, and small income companies need the kind of data to be able to process and progress.
But I am also a journalist, so that’s where my storytelling background comes from. I joined a a feminist organization that was just starting up in Asia and the Pacific as an Action Fund. For people who don’t know us, Urgent Action Fund is a set of sister funds, as we call it, which was set up in the US. There is an Urgent Action Fund in Africa, in Latin America, and in the Caribbean. We call ourselves “the younger sisters.” UAF is the brainchild of a lot of feminists coming together and realizing that Asia and Pacific need a fund that supports human rights defenders, women, and nonbinary human rights defenders in times of need.
So as the organization was being set up, one of the things that the leadership was very intentional and firm about was that the log frame kind of approach to support donors and give them change stories that they were looking for to continue funding us was not the path on which we wanted to go along. At the very outset, there was an intention that we need to look at a feminist approach to not just monitoring and evaluation, but monitoring, evaluation, accountability, and learning. For the accountability and learning part, we focused on figuring out who we were accountable to. Who should we be accountable to? Is it just the donors? Who are traditional monitoring and evaluation log frames usually in service of? Or should we also be accountable and share our learnings with the movements and organizations that we work with and work for? So that’s where the genesis of this using emergent learning as a process came through. With one of my mentors, Mikhail Friedman, is also on this call, and I and our colleagues, we have literally done this journey together to kind of figure out – what would it take? So that’s how we usually frame any question of inquiry. What would it take to support defenders and organizations through times of crisis, through their resistance to resilience?
At the very outset, there was an intention that we need to look at a feminist approach to not just monitoring and evaluation, but monitoring, evaluation, accountability, and learning. For the accountability and learning part, we focused on figuring out who we were accountable to. Who should we be accountable to? Is it just the donors? Who are traditional monitoring and evaluation log frames usually in service of? Or should we also be accountable and share our learnings with the movements and organizations that we work with and work for?
Barry: The Power of Stories is well known in reading your materials. One of the things that really struck me as very important and critical to your approach was the decision about what kind of principles you were going to adopt. The lens through which you were going to look at the world. I’d like to hear a bit more about that. There’s another thing which really struck me, which is very important, and some find it almost impossible to deal with is the issue of the complexity of the issues that you are facing. So, on the one hand, you have this series of strong anti-patriarchal principles where the human flourishing of everyone is what drives you. Then you have got this external environment which is extremely complex and hard to understand. How does the way you approach things bridge between the issues of the principles that you are looking at the world through and the complexity of the external environment, which is often hostile towards those principles?
Deepthy: That’s an interesting way of framing that, for sure. What really appealed to me when I was reading Measuring What Matters recently was how I could do almost a checklist of everything that you all crowdsourced and put together and what we had done. So, we did it retrospectively. I was looking at the paper and seeing how it was the points you put out. There needs to be motivation to measure, which was what I was talking about, leadership. There needs to be capacity to measure, which was why I was intentionally brought in with what we would measure.
The best part of creating an alternate approach for me is the collective accountability that it builds. There’s a lot of effort involved, and I think that’s where it becomes tough for smaller organizations which are heavily reliant on funders and what they seek from them as outputs. I understand the difficulty involved because any alternate processes, I think, are facilitated, and externally driven. You need an external eye. You need a 20,000ft perspective telling you, what are you sitting on? For us as a funder like us, we have moved from COVID-19 as a crisis through to Afghanistan, the Myanmar coup, Sri Lanka political turmoil, all of those are happening, not one after the other, where we are taking lessons from one and pulling into the other, but it’s happening simultaneously. So very often when we talk to each other, the way we talk about it is to imagine you are sitting in one of those Japanese bullet trains and everything rising past, but how do you keep that sense of stability within? How are you able to look at what’s happening and still derive a sense of, okay, this is the context we are working in. We still need to have a measure of collective care in place to be able to figure out how to respond with care and with the resources that the communities that we work with need. So, setting it up on values was important. But another important thing, Barry, is about how committed are we to the process?
The best part of creating an alternate approach for me is the collective accountability that it builds.
The situation that we would have set out our intent and goal for the program would not be the situation even six months down the line. So how do we make that shift? What do we rely on? If you say that you need data before you can shift, then you’re not giving true credence to the intuition, the collective intuition of the team. It’s not just about the intuition of one person and do we go with one person’s idea of how we should go? But if we were to collectively decide based on what we are hearing from the ground, what we are realizing is that robustness of the process sounds as in good shape as we move along.
Barry: It’s clear you put a great deal of emphasis on activity, building common answers to problems within the culture of difference because people within organizations will have different perspectives. How do your funders respond to this kind of approach? Do they like it or do they find it unusual?
Deepthy: Well, it started as unusual, and it started because we always talk about the experiments we are doing. We have yet to find a funder who says, no, you need to work on our log frames. Not yet. Even bilateral donors that we are working with are very interested because what I understand is there’s rampant dissatisfaction with how we are measuring and it’s about the inertia to change. You need to build that thought. You need to build that momentum to be able to shift and move into another process. I think there’s a lot of interest, which is what I see even as we publish our approach. To be honest, we didn’t arrive at it knowing that this is how it would pan out. We started with having silos. We reflected on why those silos are.
Why do people feel like they are being attacked when questions are being asked as to why this is not happening? So, everything that we’ve evolved, the point that we are at right now is a point we have fought very hard for to arrive at. So, we identified problems as they were happening. We named them. We created spaces where we could have discussions around it. And what emerged was discussions, strong discussions around power. Whose story are we prioritizing? How do we practice flexibility? How do we tell stories so that our setbacks or our failures are always learning. So, what we realize is everybody is interested in the story. It’s just that as donors and as grantees, we don’t have a platform to discuss what are the stories that matter to you? What is it that we are doing that would resonate with you?
There are fewer platforms, where we can do that kind of discussion with that bilateral funder. We had these conversations where we brought in our stories, and we looked at what they needed. Because ultimately, even at the funder level, they’re not trying to impose their log frames without a reason or not. So, we try to go to the underlying factors of what is it that would convince the committee that they need to respond to and that this investment is powerful and truly has the potential to generate impact. I think I need to be very careful about not falling into that jargon of impactful language. So, I think collectively framing those definitions was what has been succeeding for us so far.
Barry: That approach is what I love because the framing question that you come up with: what would it take to whatever you put in that next year? What would it take to shift the power towards we can all finish those templates in ways that we think are important. It’s eminently practical, whereas it’s interesting how the evaluation industry is obsessed with the idea of what is true and what is not true. What really matters is what works to get the kinds of societies we want. What matters is how people feel about the changes that are taking place rather than whether they increase a metric like GDP or something like that. Is that a kind of fair representation of what you’re saying?
Deepthy: I’m mindful that I’m talking from the perspective of a small fund, roughly a budget of USD 5 million as compared to where we are talking about GDP. I do feel that we are talking about communities. We are talking about the smaller changes which then can be scaled up if there is collective consensus and collaboration. What you were talking about in terms of everything, when you take it to design level thinking and then you make it into a template, you lose out on the nuance that gives colour to the story. By colour I mean the blood that runs through the waves. We kind of dissect it so much that in isolation, that story does not make sense anymore. That’s the disconnect I often feel when I see case studies in annual reports or other reports which the donor has put out as a success of impact. It’s often something that you can dissect and say, how is this impact? How are we looking at it as change? Why does the community feel it’s a big change. We have lost out on that context, and we have lost out on that cultural nuance.
Barry: From that we should avoid the idea of having some kind of universal template that can fit everything, but it does seem to me that if we could scale up the understanding of what you and others like you are doing, we could change the way the monitoring and evaluation industry is working. Do you think that’s a possibility?
Deepthy: Well, I think it’s possible because you and Dona wrote Measuring What Matters with a series of 136 people, unbeknownst to you and us, were doing something similar because the need felt the same. Then when you compare our papers and approaches, we see how similar they are. It basically means that the collective wisdom is showing us that this is the way to work. What we need is maybe a lot of us sitting together and trying to say that we will do the heavy lifting that’s required to change the way things are done and make it intuitive rather than counterintuitive as things have become.
Barry: Because none of us is as smart as all of us, we can use our collective intuition to begin to build a new, different kind of way of seeing not a model, because a model is something that is fixed and would violate your comments around not having a template. If we could begin to see that different strands coming together can build a picture of complexity that fits with an evaluation model that suits #ShiftThePower, because shifting power is coming from all sorts of different angles and there is no single template for that. So, I think this is a point we might begin to bring in more people.
Audience Question: I am also really interested to know the things that are hard to measure, like the way people feel and how those changes over time and people’s relationships and attitudes, beliefs, perspectives, sentiment, even the idea of resilience.
Deepthy: We’re working on that right now. I am working on a program called Shifting Narrative. One of the dilemmas that we are sitting on here is how do we know what narratives have shifted? What is the time frame that is sufficient to say that? I think starting with something like saying we don’t know, we are experimenting with this. So as a fund, we are very intentional about censoring care in all our approaches. We have convenings where we bring in defenders and organizations. We have conversations that are like this, as we are ask, what are we feeling? What is the sense right now across communities?
What we realize is that cross pollination happens when we bring different communities together and allow them to hear each other’s stories, that gives a different energy and that also results in a lot of stories that then these communities share in their own way. What we usually do is instead of figuring out how to measure them, what we try to do is build relationships of trust with the donors, and the communities they are ultimately giving money to via us. We try to have communities where all these groups are brought in. We rely then on the age-old traditions of storytelling and facilitated sessions to bring about these stories, rather than thinking about how we create indicators that we measure. As I’m saying this, we still have a five-year-old fund as we go along. When we are looking at setting indicators for shifting narratives, I think your question will be very relevant to us.
Audience Question: I was just interested to hear about how they (UAF) use art in the program and at what stage they start using the art. I would like to hear more about that.
Deepthy: We use art across the board. It started as something which we felt was a unifier, one that allowed us to take our messages across context, cultures, and linguistic divides with very little lost. If you look at our website, we have a lot of cultural influences, languages that we have used there. We have tried to make it as accessible as possible because ultimately the power we felt was in diversity. We use art across our reports. We use art for our convenience. We ensure that there is always a creative facilitator who’s bringing different modalities in, especially since we are aware that most of the communities, organizations, activists, and defenders that we work with are either in crisis or have come out of a certain amount of crisis. Art is used as a therapeutic and collective care tool as well.
Increasingly, we are using it in our system narrative program. What we realize is that especially when we use art in convenience, that visual documentation then becomes a tool when they are facilitating in their own visual spaces. Something that they experience in a convening that we did. I will give you an example. We did a convening with about 30 activists and defenders across Asia and the Pacific. What we realized out of it was we pulled together all the stories that they shared with us, and we created an infographic on what resilience means. We kind of mapped it across the different kind of practices, thinking, philosophies and different approaches people took. That infographic grantee from Sri Lanka, sent us a photograph later where they were using it as a resource to help their activists understand that these are all ways in which we could explore resilience collectively and that then became a workshop tool for them. We have been trying to see different ways in which art can be a bridge between different contexts and culture, but also between philanthropy. That’s our new hypothesis. How do we use art and the language of art from communities to be a bridge to influence philanthropy?
Barry: What I am learning from this discussion is that we can, if we are really creative in the way that we are sending data out here, actually integrate scientific approaches because you are collecting data from your grantees and grantee partners, and you are also using stories but also visual material. We can begin to get some kind of new synthesis, which is not either or. Either things are scientific, or they are artistic. We are beginning to get a rounded view of truth, which is multidimensional, which I think is a critical point if we really want to measure what matters, because we need to articulate the different dimensions rather than going towards a singular goal. The inventor of GDP, the economist Simon Kuznets, said, whatever you do, you must not use this measure-to-measure human happiness, because it’s not a technocratic economic measure. Yet we have tended through the 75 plus years of international development to equate economic development with being well off, and it is not the same. Our mind horizons in these kinds of ways if we are going to make progress. And there is no simple formula to do that because an algorithm will always be limited because it rules out certain things. So we need a different kind of imagination mindset to be able to build the kinds of societies we want. That’s where this discussion is taking you.
That’s very clear from this discussion is that changing paradigms measurement doesn’t mean that we throw rigour out the window. We just need a wider sense of what rigour really means because there is no single method, no single kind of approach which will give us the complexity, that gives us wider pictures. One of the commonest problems is how do you measure soft variables like trust? We really need trust as a premium in the world and we need to put that into everything we do, and we need to find good ways of testing that. Ticking boxes is not necessarily the right way to do it is one of the ways, but without the kind of richness of the experience, context is important.
If you liked this, you might also like: