Sustainable Development Goals: evaluating progress for a brighter future – podcast
Every country has 2030 as the date for achieving the Sustainable Development Goals (SDGs). Some are well on the way to meeting targets, others have further to go. In this new three-part mini-series, we explore how to evaluate progress against the goals.
Evaluating how well countries are meeting targets requires work across different sectors and themes. It’s not only a question of how to do the evaluation, but also, once the results are in, how to support government departments and civil society to act on the findings.
And as countries deal with the impacts of climate change, including assessment of how climate risks have affected SDG strategies and may do so in the future has become an essential part of any evaluation exercise – but the knowledge of how to do that effectively is not widespread.
This podcast series draws on the knowledge and learning from a longstanding partnership between EvalSDGs (a network of development institutions, policymakers and practitioners), the German Institute for Development Evaluation (DEval) and IIED, with funding contributed by EvalSDGS through its membership of EvalPartners.
Over the course of three episodes, hosts from each organisation discuss the details and challenges of assessing progress against the Sustainable Development Goals (SDGs) with other specialists in evaluation theory and practice.
The podcast covers:
- Episode 1: Adaptive evaluation: considering climate risks in theory and practice
- Episode 2: Defining and scoping an SDG evaluation – how do you get started?, and
- Episode 3: Country-led evaluations: revealing the progress of Agenda 2030.
Subscribe
You can subscribe to the podcast on your favourite podcast app as follows:
The podcast is also available on IIED's YouTube channel.
Episode 1
Adaptive evaluation: considering climate risks in theory and practice
In this first episode, IIED hosts a discussion about the principles, practice and challenges of considering climate risks in evaluation of programmes aimed at achieving the SDGs.
The conversation features:
- Stefano D’Errico, head of monitoring, evaluation and learning in IIED's Strategy and Learning group
- Emilie Beauchamp, senior researcher in IIED's Strategy and Learning group
- Nick Brooks, director of Garama 3C and adaptation specialist
- Samson Machuka, director of the monitoring and evaluation department in the State Department for Planning and Statistics of the Government of Kenya.
Episode 2
Defining and scoping an SDG evaluation: how do you get started?
In the second episode, produced in both English and Spanish, DEval hosts lead a discussion on the elements to consider for getting started on an SDG evaluation.
The conversation in English features:
- Carolina Zúñiga Zamora, analyst in the evaluation unit for the Ministry of National Planning and Economic Policy of Costa Rica
- Satu Lähteenoja, senior expert on sustainable development at Demos Helsinki
- Gonzalo Hernández Licona, senior research fellow at the International Initiative for Impact Evaluation (3ie)
- Dirk Hoffmann, evaluation expert from DEval
Listen to our Make Change Happen podcast, which provides informal insights into IIED’s work to create positive change and make the complex issues we face more accessible to wider audiences.
Episode 2
Defining and scoping an SDG evaluation: how do you get started?
In the second episode, produced in both English and Spanish, DEval hosts lead a discussion on the elements to consider for getting started on an SDG evaluation.
The conversation in English features:
- Carolina Zúñiga Zamora, analyst in the evaluation unit for the Ministry of National Planning and Economic Policy of Costa Rica
- Satu Lähteenoja, senior expert on sustainable development at Demos Helsinki
- Gonzalo Hernández Licona, senior research fellow at the International Initiative for Impact Evaluation (3ie)
- Dirk Hoffmann, evaluation expert from DEval
Defining and scoping an SDG evaluation: how do you get started? full transcript
Dirk Hoffman [00:00:13] Hello. Welcome to episode two of the podcast we’re calling Sustainable Development Goals, evaluating progress for a brighter future. I’m Dirk Hoffman from Deval – the German Institute for Development evaluation, and our new host.
Dirk Hoffman [00:00:34] In episode one of the series, we heard how important it was to consider the effect of climate risks in the evaluation of strategies and programmes designed to make progress towards the sustainable development goals. That’s certainly critical. But how are these evaluations done? What are the best methods? And what are the potential stumbling blocks and challenges? Why have so few SDG evaluations been done up to now? That’s what we’ll consider in this episode, because we know these evaluations are complex. Only some countries have done them and in the process have learned lessons that can be shared with others.
For starters, we’ll think about how to define the scope and the focus of the evaluation. Sometimes a major challenge. We’ll hear about the first ever national evaluation of SDGs carried about by Finland in 2018 and 2019, where Finland took the bold decision to look at the whole set of all 17 SDGs together. And we’ll also hear from Costa Rica that delivered an evaluation report at the end of 2020 that had been two years in the making. And I’m glad to say that the lessons it learned have been captured in a parallel process that was carried out along with the evaluation itself. We’ll hear more about that shortly. So let’s get started. And that means I’m first going to ask my three guests to introduce themselves.
Carolina Zuniga [00:02:02] Good day everyone. My name is Carolina Zuniga and I work for the Ministry of Planning of Costa Rica, which is the steering institution regarding evaluation for the public administration. Specifically, I work in the evaluation unit. And, well, it's my pleasure to be here and be able to share our work.
Satu Lahteenoja [00:02:23] Hello, my name is Satu Lahteenoja and I'm a senior expert on sustainable development in an independent think tank called Demos Helsinki in Finland and I was leading the, the first national SDG policy evaluation in Finland. So in my work and research I am looking for different ways to accelerate sustainability transformations and I see evaluation as one of the ways to do that.
Gonzalo Hernandez Licona [00:03:01] Hello, how are you? My name is Gonzalo Hernandez Licona, I'm from Mexico and I'm the director of the Multidimensional Poverty Network and I'm also working with the Global Evaluation Initiative, Unicef, IDB and all those in terms of evaluation. I've done evaluation for the past 25 years so I'm very excited to be here. Thank you very much.
Dirk Hoffmann [00:03:24] Well thank you for those introductions. Let's start with the basics: definition of SDG evaluation. Perhaps we shall be clear what we mean when we say ‘an evaluation of the Sustainable Development Goals’ or when we talk about ‘country-led’ evaluations. Gonzalo would you like to start us off?
Gonzalo Hernandez Licona [00:03:43] OK, thank you Dirk. I believe that the first thing we need to remember is that, is what the SDGs are about. They are not only about a collection of goals to advance on them separately. The SDGs have principles. For instance, integration and coherence or leave no one behind. So, for instance in integration, what we mean on the SDGs is we have to advance on, on the objectives and the targets together, right? So there the SDGs emphasises the interlinkages between goals and targets. Sometimes the trade-offs between the dimensions.
So, it is important that an evaluation of SDGs should follow those, those principles. And for instance, to address objectives at the same time, so give an example: even in a family we need to move together with education, health and nutrition otherwise a child cannot go to school if those three elements are not addressed at the same time. It means that we need to evaluate in that case that the family is moving on those objectives together. The same is happening in the evaluation of SDGs - we need to move forward but taking into account the advances in all of them at the same, at the same time.
Dirk Hoffmann [00:05:06] OK. Now how many SDGs does it take to make for an SDG evaluation. Do we have to look at all 17 SDGs or is it it enough to just look at a single SDG? What is the experience of Costa Rica?
Carolina Zuniga [00:05:23] Well that’s a tough question, especially for us because I don't specifically think that I have an answer for that, because I don't think there's a magic number. We have this conversation in our institution, especially in our evaluation team when we were talking about the next national evaluation agenda. And we came to the conclusion that it’s necessary for us to identify what is specifically the information that we need. And well in that case, it's not about how many SDGs we're going to evaluate, it’s more about what else we need to know regarding the process of the sustainable development.
And in our case, we haven't performed like a specific SDG evaluation like Nigeria did or like Finland did. But we do perform country-led evaluation with an SDG component and we include this component in every evaluation since 2019. And for us, it's very important to identify what is it specifically that we need to know and then evaluating that specific component. In this case, we're talking about the biodiversity and climate change evaluation, which is one of the evaluation inside our national evaluation agenda, with a broad analysis of the SDGs. So I don't think there's any specific number and I completely agree with what Gonzalo just said: it’s more important to identify how, are we going forward in the sustainable development?
Dirk Hoffmann [00:07:13] Thinking about what you just said, how does evaluation relate to the SDG policy cycle? Gonzalo, what would be your take?
Gonzalo Hernandez Licona [00:07:21] So, let me go back a bit of what Carolina said rightly, I mean there's no magic number about how many goals should we take into account to name it an SDG evaluation. I think as long as it reflects that we are evaluating systems, and as long as we include in this the policy cycle, right? Which means we have to understand that our evaluations should improve what we do. So, and as long as we include some of the principles like integration, leave no one behind, resilience, universality, right? So for instance, if we want to, to evaluate the No Hunger, SDG number two objective.
So we need to assess the various elements that take us to have zero hunger. We need to assess the food production, waste, consumption partners, nutrition markets, green solutions to the food processes. So we have to evaluate the policy of zero hunger instead of one single programme, and taking into account that we need to put it in the cycle of the policy in the government, meaning that many actors should improve, according to the evaluation that we are producing.
Dirk Hoffmann [00:08:44] That sounds very complex, taking into account the policy and all the actors involved. If we just think about one SDG, Finland has opted to look at all 17 SDGs at the same time. How did you tackle this challenge of having 17 SDGs to look at, so many sectors to look at and, and all the different stakeholders?
Satu Lahteenoja [00:09:08] Yes, in Finland we really did look at all 17 SDGs because we wanted to have a comprehensive evaluation. So having an overview of where the country is and then it didn't make sense to leave something out. However, we didn't look at all 17 SDGs with the same depth. And I think one of the key challenge for our evaluation in Finland was like how to cover the sort of key substance issues in reasonable depth with restricted resources, and I think this is one of the general concern in sustainability evaluations when the topics are often very broad. In our case, the solution was to find the top five sort of most burning sustainability concerns by indicator analyses, surveys, expert interviews. We then found that competence and societal stability were sort of, one of the strengths of the country and then there were many, like, systemic needs for improvement, including energy system, use of forests and and the global footprint of our consumption, as examples.
Dirk Hoffmann [00:10:51] OK, so you clearly went for a prioritisation of the SDGs and looked at which ones are most relevant to the country. We've just heard Gonzalo talk about the SDG Agenda 2030 principles, that they are important. Like he mentioned coherence, equity, participation, leave no one behind. How did Finland cater to the principles underlying the 2030 Agenda? Or did you just... or did you basically look at the targets and the interactions within the 17 SDGs?
Satu Lahteenoja [00:11:31] I think it's very reasonable to evaluate the policy against its principles and priorities. And I think this - the 2030 Agenda principles - can also work well as an evaluation criteria. In the case of Finland, we use the focus areas and policy principles of Finland as the criteria as Finland had defined, sort of, own sustainable development policy principles. And those have, of course, a lot common with the 2030 Agenda principles. So they include like long-term action and transformation, then policy coherence and global partnership. And thirdly, commitment and participation. So in our evaluation we used those as criteria, asking basically are the policies coherent? Is our sustainable development policy creating long-term transformation? And so on. Of course, super, super big questions in one evaluation.
Dirk Hoffmann [00:13:05] It is always difficult to tackle SDG evaluation because it's such a broad topic. It seems so broad. Let me move into the next set of questions looking at what is the value of doing an SDG evaluation and you've already talked a little bit about it, Satu. Within the UN Economic and Social Council, the EcoSoc, a reporting mechanism has been set up for the SDGs. Know that each country is asked to report on the state of implementation of the 2030 Agenda in what is called the national voluntary reviews - or VNRs for short. Gonzalo, about the importance of doing SDG evaluations what can you tell us in relation to the VNRs? I mean how is evaluation coming into the exercise of doing voluntary national reviews?
Gonzalo Hernandez Licona [00:14:06] Oh, OK, so we just mentioned a little bit of that before that it is important to evaluate the forest and not only the tree or the leaf - we have to see the whole picture. I mean development is complex, Dirk. If development was only about cash transfer programmes then development would have been easy. But development is about incentives to grow and to innovate, it’s about balance of power between agents. It's about addressing various dimensions. And therefore, while we, I believe we need to do, when we write and we produce VNRs – voluntary national reports - is that of course we make accountable of what we're doing in the country. We describe what is happening in the country in terms of SDGs. But we also have to be critical about what we're doing, right? Is, it may be common that VNRs look very, very nice [laughs] because sometimes the countries put that, that everything is wonderful. And therefore we in a very, in a true VNR, we should put our advances, the full picture of the development that we are facing in the country, as well as the challenges so we all could learn and we all could, could move forward with the advances and challenges.
Dirk Hoffmann [00:15:26] Let, let me ask the same question to you, Carolina, for the evaluation that you did in Costa Rica. Why do we need evaluation when we have already voluntary national reports and Costa Rica has filed voluntary national reports already? What about the relation of the VNR with, with the evaluation of SDGs from your point of view?
Carolina Zuniga [00:15:50] Well, in our case, we see it more like a process because for sure, evaluation is an input for the VNRs. And the VNRs are not only for accountability, but to ask ourself if, are we on the right path? What else is needed to achieve our commitments? And for example, in our case, we have been identifying as part of the several evaluation findings that programmes and projects, strategies, they all need to design considering the SDGs because otherwise it will be hard in the future to achieve them. And overall we see an opportunity because the VNR becomes another input also to identify priorities in evaluation. In our case, we are at the door of a new government. This means a new evaluation agenda and we have to ponder what needs to be evaluated. Is it a specific SDG? Is it a complete - as Satu said before - like a complete policy regarding a specific SDG? We don't know but we need to identify what is most important for us. And that is information that the VNR can definitely give us. So we see it as a cycle: evaluation is an input but also is the voluntary national reports.
Dirk Hoffmann [00:17:21] That, that seems to me very interesting that the movement goes both ways. Evaluation ideally informing the VNRs and the VNRs informing SDG evaluation. Would you agree on that or how does that work in practice, Satu?
Satu Lahteenoja [00:17:41] Yes I very much agree with the others and that's how it, it goes also in Finland. And what sort of, what inspires me is that with the help of an evaluation we can really have an overview of where we are in terms of SDG implementation and then understand that, where the challenges are, and then based on that and that is often not done in a VNR, is to develop then recommendations on how do we then come closer to reaching the goals. And to get there, in my opinion, that evaluation should be at the same time out of participatory developmental and future oriented. Or at least all of those were needed in the Finnish case and there of course, VNR is one important tool in there.
Gonzalo Hernandez Licona [00:18:57] Dirk, can I add something else?
Dirk Hoffmann [00:18:58] Sure, go ahead.
Gonzalo Hernandez Licona [00:19:00] You know I think taking the examples of Finland and Costa and Costa Rica for the VNRs is very important because they include evaluation processes in the VNRs. They include challenges in the VNRs. And I think that's very important because, as I said before, where you put a VNR, which everything, where everything is really beautiful in the country, it is not the way we have to do it, you know what I mean? So example of these countries are very important.
Dirk Hoffmann [00:19:27] OK, I think that is an excellent point. Thanks for signing this to us. We've heard a lot about the benefits of doing an SDG evaluation. So what strikes me a little bit is the discrepancy with the very few evaluations on SDGs that have actually been done. We're now already seven years into the 2030 Agenda so what seems to be the major difficulty? Is it finding the right start that is like a major hurdle? Would you agree on that, Gonzalo?
Gonzalo Hernandez Licona [00:20:00] Well, I believe because as I said development is complex. The agenda is comprehensive and therefore complex. We were used, or used, to evaluate single projects or single programmes, so from going to evaluating a single programme to evaluating a whole full agenda - that's, that's tough. But I believe that, that countries we should carry on in trying to do this with simple questions where, for instance: how did the concept of development change in the country as a result of the agenda? What happened in the country when this SDG arrived? What was the change in terms of development in the development strategy? What is the sustainable development theory of change in your country? Does the country have a solid data generation process and institutions?
And a very important question that I believe is - I mean it's a tough one but we have to address it - is how are the national priorities made compatible with the 2030 agenda? And I think there's a key, as a very key element, how we, how the countries combining the priorities that that already the country have with a new SDG agenda. I think addressing that question with clarity, it’s a good start for instance.
Dirk Hoffmann [00:21:32] So, aligning national priorities to the SDG Agenda you tell us that is the most important thing we have to focus on? Carolina, taking your experience with the evaluation in Costa Rica does that ring true? I mean you've actually done an evaluation. How did you get started?
Carolina Zuniga [00:21:50] Yeah, definitely. For us it was exactly like that [laughs]. Because we were the first country in the world, right, to reaffirm a high level collective commitment to achieve the SDGs. And for us this meant put the 2030 Agenda as a country priority. And however we include this specific topic into evaluations until 2018. Of course there was a lot of management to do between the identifying of how to measure the indicators, where to get the information, setting governments - governance, sorry.
But for a moment I think that we forgot about evaluation and the SDGs and I think it's just like, like Gonzalo just said, it's how we combine the priorities of the country with the international commitments. And we decide the evaluation, with the biggest component regarding the SDGs was going to be probably the most complex evaluation inside our national evaluation agenda. So it's not easy for sure. It's not easy, but it is about make it happen. It is about what is important.
Dirk Hoffmann [00:23:10] Yeah, this is to me a very important point. You have to get started. You can't wait for everything to fall in place and then move ahead. You, you have to identify national priorities. You have to take decisions and as Satu earlier explained, expert interviews, etc, involving stakeholders and then moving ahead.
Dirk Hoffman [00:23:42] I would like to ask in the concrete case of Finland because we have the country experience, Satu. Can you give us an, as an example, how did you go about defining scope and focus of the evaluations?
Satu Lahteenoja [00:23:56] In our Finnish case the focus and scope of the evaluation was quite broad and it was defined in the, in terms of reference. And that already then included also the, the evaluation questions. However, we then shaped those together with the evaluation team and with the steering committee when we started to work. And if I like very briefly reflect on the choices we did, I think this concentrating on these like certain sustainability issues instead of trying to cover all in equal depth was a good choice. But then, however, in addition to detecting on problems I think we could have given more attention also to strengths and, and in particular like teams and questions in which, like a clear sustainability contribution could be made also globally. So, so by asking, like, what is the global sustainability handprint of your country and our, or what it could be, that your country contributes to the to the global sustainability?
Dirk Hoffmann [00:25:34] Ok, I think that is an aspect that is maybe not the case with each and every SDG evaluation, putting the country in a global context. You know, usually from what I've seen it works the other way around, that global context is the SDGs and that is put into a national context. So that's interesting your perspective that you're giving us. Something else that we've been talking about a little bit already mentioning here and there but I think deserves a little bit more attention is the participatory process that an SDG evaluation might be, or should be. How do we go about designing a participatory process? How far was it, the evaluation participatory in Finland?
Satu Lahteenoja [00:26:27] Yes, as we all know these participatory processes, they tend to take a lot of time. But I would say it is really crucial. So in them, in the Finnish case we had like several groups we wanted to involve into evaluation. We had a steering group with representatives from the ministries and then we had, like a support group with like 20 experts from different fields: sustainability, evaluation experts, civil society. Then we had international experts and this was very, very relevant and useful. And then we had like a large and open sustainable development stakeholder community. And then of course the Finnish parliament, especially they're the Committee of the Future. So, what we did was actually quite a lot of interviews, workshops, different discussion events and one thing that worked really well was that we co-designed the recommendations together with about 40 stakeholders and this was a way to really engage and also empower them. So, when the evaluation report was published there were actually no surprises for this group of people anymore and they were then also highly engaged to disseminate and work further with the recommendations.
Dirk Hoffmann [00:28:22] OK, this is a very valuable point which probably holds true for evaluations in general involving stakeholders in the whole process in the design and as you just said in the formulation of the recommendations. Gonzalo, from a more theoretical point of view would you see any specific challenges for participation when design an evaluation of the SDGs as compared to a regular evaluation?
Gonzalo Hernandez Licona [00:28:52] No. I believe that because we are trying to deal with, not only with a single programme or a single project evaluation, but we are dealing trying to address the full agenda. Of course, we need more people and we need more stakeholders and we need them to be on board on the evaluation and to be frank and to be, and to know that they will benefit as well for the evaluation. And that is not that easy, right? So I'm really impressed with what Nigeria, Finland, Costa Rica have been doing in terms of evaluating the SDGs because that implies a lot of participatory process.
Dirk Hoffmann [00:29:37] OK, excellent. There is another topic often forgotten when we talk about the design phase of an evaluation and that is communication. Should communication already be considered in the design phase? What is your opinion or is it better to first see the results, the recommendations, and on that basis define a communication strategy? What are your experiences in, in Finland and Costa Rica? Carolina maybe you, you go first.
Carolina Zuniga [00:30:07] Sure. I think that you can have any specific strategy depending on the evaluation sure. But in our case, this is kind of standardised already because we evaluate public interventions, and it is our job to share the final results with anyone who's interested. So yes, in our case, we pretty much do at least three things for every evaluation: we first upload the final report in our web page. And of course that we encourage the institutions involved to do the same. We also organise at least three final presentation of the results of the evaluation, considering different target population. And at least one of those presentations should be like an open one to be able to invite different stakeholders, especially citizenship, because it's very important for us for people to know what we're doing and to use also the evaluation results.
And we also develop different materials - this is different for every evaluation. But, for example, we do different videos, documents, infographics this kind of depends on the evaluation. But it's usually some kind of material that can be distributed easily. And we participate in workshops. We also attend to a specific request to share the results. We participate also in conferences because it is important for people to know that these kind of evaluations are being performed.
Dirk Hoffmann [00:32:06] In Finland did you consider communication as important in the moment when you designed the SDG evaluation or did that come in later?
Satu Lahteenoja [00:32:15] Yes I think it's very important to think about it in the design phase already. So first I thin timing is important. So, in our case, the timing of the evaluation was good as the results were published right under the parliamentary elections, so there was interest towards the results directly. Secondly what Carolina also mentioned are these visualisations and key numbers are very important in communication. So it's good to require them and also leave resources for that. And then thirdly, I think it's very important to leave time and resources for communication in general after the publication of the results. In Finland we then as an evaluation team went to discuss the results to the parliament I counted in total eight times.
So we had five different separate discussions with different political groups and three different committee meetings. Even several committees together which doesn't happen that often. And we were then also invited to present our findings in the government negotiations and I guess our work had at least a small contribution to the fact that actually the government programme of the current government in Finland is based on sustainable development. So we are at the moment quite lucky.
Dirk Hoffmann [00:34:22] Congratulations to Finland. I think this is really a great example for others to follow along.
Dirk Hoffman [00:34:36] We are almost coming to the end of this episode and maybe wrapping up a little bit, looking at the different topics. And obviously we could have delved into much, much further and this is just starting off discussion on those topics. But maybe instead of me giving overall conclusions which would be very, very difficult, I would like to do a final round briefly giving the microphone to each one of you for one main message to formulate a recommendation or one central thought for our audience. If you would start, Carolina.
Carolina Zuniga [00:35:12] I would like to bring a topic that we talked about at the beginning and throughout the whole episode and it's about the principles of the 2030 Agenda. I think that Gonzalo talked to us a lot about this. But we, present times with corona [virus] showed us that we need to have them present at all times. Not only for evaluation. But for planification, for planning any kind of intervention. Also, we have to have them present if we want to achieve common goals as a country. During the whole policy cycles and our, of course in our work as evaluators we need to know how are we going forward, if we're going in the right way. We need to share experiences, resources, knowledge, making sure that we're going in the same direction, working together. I think it's important to have them present.
Dirk Hoffmann [00:36:12] Moving over to Finland. Satu, what would be your main message, main thought?
Satu Lahteenoja [00:36:20] Quite difficult to, to choose only one main message. But perhaps for this audience I would highly encourage you to use external experts. Meaning also your experts and contacts from other countries. It helped us a lot and I think raised the level also. So use your networks to get more perspective.
Dirk Hoffmann [00:36:54] Thank you very much. Last but not least we're moving over to Mexico – Gonzalo.
Gonzalo Hernandez Licano [00:36:59] Thank you, thank you, Dirk. I believe that having evaluations beyond single programmes and single projects is a key element for SDGs. I mean you have to evaluate policies, we have to address objectives. At the same time when possible we need to do it in a participatory way. We need to look beyond the obvious I believe. I mean for instance, it's very simple, it's very common that countries just link what they were doing before with SDGs and they say, hey, we are we are addressing STGs because everything that we used to do is linked. We have to address that and challenge a little bit the country and challenge a little bit the government saying, hey, we need something else in terms of a sustainable agenda if we really would like to put it forward. And the evaluation can help in understanding that. So, so thank you very much, Dirk, Satu and Carolina to be, I'm glad that that we're here together.
Dirk Hoffmann [00:38:02] Well thank you very much from my side to, to our three guests. To Carolina Zuniga from the Ministry of National Planning and Economic Policy in Costa Rica. To you Gonzalo Hernandez Licona, Senior Research Fellow at the International Initiative for Impact Evaluation – 3IEE. And also to you, Satu Lahteenoja, Senior Expert on Sustainable Development of Demos Helsinki, independent thinktank from, from Finland. Then I would like to finish with a reference to the website, the IIED website where you can pick up additional information linked to the organisations’ websites, reference material to delve deeper into the topics we've just been discussing.
And as a last point, finish with a look ahead at episode three, the final episode of this podcast series. In this episode. We will be covering the aspects of implementation and use of SDG evaluations in order to accelerate progress for reaching the SDGs by 2030. I hope you'll join us. Thank you very much and goodbye.
The conversation in Spanish features:
- Ericka Valerio, evaluation commissioner of the evaluation unit for the Ministry of National Planning and Economic Policy of Costa Rica
- Victoria Sánchez, vice president of Red2Red
- Ada Ocampo, president of IDEAS International
- Nataly Salas Rodríguez, specialist in capacity development in evaluation for the DEval’s Focelac+
Definición y alcance de una evaluación de ODS ¿por dónde empezamos? transcripción
Nataly Salas [00:00:13] Hola, bienvenidas y bienvenidos al episodio dos del «podcast» que llamamos Objetivos de Desarrollo Sostenible, evaluando el progreso para un futuro más brillante. Soy Nataly Salas, del Instituto Alemán de Evaluación de la Cooperación al Desarrollo, DEval, y hoy seré su anfitriona. En el episodio uno de esta serie escuchamos lo importante que es considerar el efecto de los riesgos climáticos, en la evaluación de estrategias y programas diseñados para avanzar hacia los objetivos de desarrollo sostenible. Esto sin duda importante, pero ¿cómo se realizan estas evaluaciones? ¿Cuáles son los métodos más apropiados? ¿Cuáles son los posibles obstáculos y desafíos de llevarlas a cabo? Eso es lo que consideramos en este episodio, porque sabemos que este tipo de evaluaciones son complejas. Algunos países las han desarrollado ya, y en el proceso han aprendido lecciones que pueden ser compartidas con otros actores con interés en el tema.
Nataly Salas [00:01:09] Entonces, para empezar, pensaremos en cómo definir el alcance y el enfoque de la evaluación, muchas veces un desafío determinante para el proceso evaluativo. Escucharemos cómo Costa Rica culminó una evaluación de este tipo a finales del 2020, lo cual tomó dos años para completarse. Y además, me complace decir que las lecciones aprendidas de este proceso en Costa Rica, se han plasmado en una sistematización de aprendizajes, que se llevó a cabo en paralelo a la evaluación. En breve escucharemos más sobre eso. Además, tendremos la oportunidad de conocer algunos aprendizajes de otros países, que también desarrollaron evaluaciones de los ODS en otros dos continentes. Así que comencemos. Ahora le voy a pedir a mis invitadas de hoy que se presenten.
Ada Ocampo [00:01:57] Buenos días, buenas tardes con todos. Mi nombre es Ada Ocampo, soy la presidenta de la red global IDEAS, el sitio web es ideas-global.org. Para mí es un enorme placer estar en esta discusión y aportar desde mi perspectiva y mi experiencia, y mil gracias por la invitación.
Ericka Valerio [00:02:22] Hola, mi nombre es Ericka Valerio, yo soy economista con 10 años de experiencia en temas de evaluación. Actualmente trabajo en la Unidad de Evaluación del Ministerio de Planificación Nacional y Política Económica de Costa Rica, específicamente en la Unidad de Evaluación. Esta unidad es la responsable del desarrollo de la agenda nacional de evaluaciones que tiene el país. Muchas gracias por la invitación.
Victoria Sánchez [00:02:47] Muchas gracias por mi parte también, a Nataly y a las organizaciones que están promoviendo este encuentro tan interesante. Mi nombre es Victoria Sánchez Esteban y soy vicepresidenta de Red2Red, que es una empresa española de consultoría. Y durante los últimos 25 años, aproximadamente, he estado trabajando como consultora del sector público y fundamentalmente en el ámbito de la evaluación de las políticas públicas, con mucha frecuencia en América Latina, sobre todo en los últimos 18 años aproximadamente. Y luego en lo concreto, tuve la competencia de coordinar la ejecución de la evaluación que mencionaba Ericka, que se llevó a cabo en Costa Rica, la Evaluación de la Cooperación Internacional No Reembolsable en Biodiversidad y Cambio Climático. Muchas gracias de nuevo.
Nataly Salas [00:03:38] Gracias a ustedes, Ada, Ericka y Victoria por acompañarnos en este interesante «podcast». Comencemos con lo básico, para tenerlo claro, ¿qué entendemos cuando decimos una evaluación de los Objetivos de Desarrollo Sostenible? Ada, ¿qué piensas?
Ada Ocampo [00:03:54] Gracias, Nataly. Una evaluación de las ODS, yo quisiera ponerla primero en perspectiva. Como todos recordarán, la Agenda 2030 tiene cuatro capítulos, siendo el cuarto capítulo el de Seguimiento y Evaluación. En ella, a través de la agenda y específicamente ese capítulo, los gobiernos se comprometen a generar evidencias a través de evaluaciones y sistemas de monitoreo, para reportar y analizar su progreso hacia los ODS. En ese marco, hay una responsabilidad de parte de los gobiernos que firmaron este acuerdo, de avanzar en ese sentido. Pero ¿por qué es importante, más allá del compromiso? El compromiso tiene que ver con una rendición de cuentas que los gobiernos se comprometieron a brindar. Pero más allá de eso, es importante que los gobiernos, los países, conozcan en qué medida, basados en evidencia, están avanzando hacia estos ODS, y cómo este análisis a través de evaluaciones puede repercutir en mejores políticas públicas y más equitativas.
Victoria Sánchez [00:05:14] Nataly, si me permites, yo tengo una pregunta para plantear aquí al grupo que estamos conversando. Y es que en ocasiones me he preguntado, cuáles son los límites para considerar una evaluación con una evaluación de ODS. Un poco siguiendo la línea de lo que estaba mencionando Ada, cuando vas a evaluaciones que, por ejemplo, pueden ser temáticas, o vas a evaluaciones de intervenciones más específicas, a mí se me genera la duda de cuál es el límite para poderla considerar una evaluación de ODS.
Por ejemplo, si solamente nos interrogamos sobre en qué medida esa intervención está alineada con los ODS, pero solamente es una pregunta en el marco de un conjunto de preguntas de evaluación, ¿eso sería una evaluación de ODS? O si, por ejemplo, utilizamos los principios de la agenda como marco analítico junto con otros elementos de análisis, como pueden ser, por ejemplo, los criterios del CAD de la OCDE, ¿eso sería una evaluación de ODS? Y yo no tengo una respuesta, o sea yo lo planteo únicamente como una pregunta aquí al grupo para seguir hablando sobre ello o no, simplemente como mi duda.
Ada Ocampo [00:06:28] Yo quisiera reaccionar a la interesante pregunta de Victoria, para recordar que las evaluaciones y más específicamente las evaluaciones de la ODS, tienen un componente político y al mismo tiempo un componente técnico. El componente político es tan importante, o quizás en un inicio más importante que el técnico, porque la decisión política es la que va a permitir que la evaluación se realice, que cuente con el apoyo de las instancias que son importantes dentro del país, y que cuente con un presupuesto también. En ese sentido, yo quisiera ligar la pregunta de Victoria, ¿hasta qué punto se considera una evaluación de los ODS? Yo diría que eso está ligado al tipo de evidencia que se va a generar y cómo se va a utilizar.
Ada Ocampo [00:07:19] Si la evaluación genera evidencia de cómo se está caminando o cómo se puede caminar, progresar mejor hacia los ODS, cómo se puede hacer para que la estrategia en los sistemas, los planes, están más alineados con los ODS, entonces se puede decir que es un ejercicio evaluativo o una evaluación que está dentro del marco de lo que pide la agenda. Entonces, pueden haber diferentes situaciones pero que tienen que ver, por un lado, cómo estas tipo de evaluaciones se enmarcan dentro de la agenda, y en segundo lugar, y quizás más importante, cómo se intenciona el uso de la evidencia generada para dar cuenta del progreso hacia los ODS.
Victoria Sánchez [00:08:11] Genial, muchas gracias. La verdad es que me queda mucho más claro ahora, Ada.
Nataly Salas [00:08:15] Que interesante. Con lo que explica Ada, es evidente que no hay una sola manera de hacer evaluación de ODS, y que lo importante es generar la evidencia para poder dar respuesta, a ver si vamos avanzando en la consecución de la Agenda 2030 o no. Ericka, ¿cómo lo ves? ¿Cómo fue la experiencia de Costa Rica? ¿Por qué les parece o le pareció importante al Ministerio de Planificación de Costa Rica hacer una evaluación con enfoque ODS?
Ericka Valerio [00:08:45] Bueno, nosotros desde Costa Rica estamos dando los primeros pasos en evaluación de los Objetivos de Desarrollo Sostenible. Nosotros tenemos una Agenda Nacional de Evaluaciones, que sus siglas son ANE, la cual está incluida en el Plan Nacional de Desarrollo de Costa Rica. Para construir esta agenda se incluye un criterio de priorización, que está relacionado con la alineación de las intervenciones con los Objetivos de Desarrollo Sostenible. Además, en Costa Rica tomamos la decisión de incluir, en todas las evaluaciones que se hacen de la agenda, una pregunta que está relacionada también con los ODS, una pregunta de evaluación y la Agenda 2030.
Ericka Valerio [00:09:27] Con las evaluaciones que nosotros hemos desarrollado, como principales hallazgos ha sido que para involucrar a las instituciones del gobierno en el logro de la Agenda 2030, es fundamental identificar cómo estas intervenciones y sus objetivos se alinean y aportan a los ODS, y a partir de esta identificación generar cambios en lo que es política pública. Las evaluaciones nos permiten identificar si se ha logrado, qué hace falta, qué hemos hecho y qué no. Y bueno, finalmente esto se traduce en decisiones para seguir avanzando, es por eso que es importante para nosotros realizar evaluaciones de los ODS.
Nataly Salas [00:10:12] Muchas gracias, Ericka. Me parece que queda muy claro también el aporte que han tenido en Costa Rica, de las evaluaciones en las que han incluido aspectos de valoración o componentes de los ODS o de la Agenda 2030. Hablabas de cambios en política pública, que esto es fundamental para poder alinearlos con la agenda, y pensando en desarrollo sostenible a largo plazo. Victoria, ¿qué piensas? En términos más generales a nivel global, no solo a partir de la experiencia de Costa Rica. ¿Y qué retos ves también para los países en este sentido?
Victoria Sánchez [00:10:47] Sí, yo creo que un reto bastante generalizado y que obviamente depende mucho del país, y de cuál sea la profundidad y la calidad y la actualización de sus sistemas estadísticos. Pero uno de los retos principales, creo que tiene que ver con la disponibilidad de la información requerida para poder medir y contar con información, para poder valorar el avance en la dirección del logro de los ODS. Con mucha frecuencia, esta disponibilidad no es la que se desea y la que sería necesaria, con lo cual se dificulta la valoración.
Victoria Sánchez [00:11:24] Y otra cuestión, que por lo menos sí que nos encontramos en este caso, es la que tiene que ver con el alcance temporal de las intervenciones. O sea, los ODS se aprueban el 2015, se empiezan a aplicar un poco después, en función de diferentes elementos sectoriales de país, los compromisos que se alcanzan, etcétera. Pero a veces cuando evalúas, evalúas intervenciones que han comenzado antes y que han seguido funcionando, han seguido desarrollándose ya en periodo de ese, por decir así. Entonces, el captar el cómo estas intervenciones, tanto en términos de alineación como en términos de avance, puedan contribuir a estos ODS, creo que también puede ser un reto de carácter práctico en muchas ocasiones.
Nataly Salas [00:12:10] Ada, ¿quisieras agregar algo?
Ada Ocampo [00:12:12] Sí. Me parece que Victoria ha tocado los principales temas que tienen que ver con el lado técnico, y yo a esa dimensión quisiera agregarle solamente un reto adicional. Hay situaciones mayoritarias que la información no está disponible, como dijo ella, no necesariamente la información está disponible, pero cuando está disponible, no necesariamente tiene el nivel de desagregación que la agenda pide para realmente evaluar temas de equidad. No necesariamente la información está desagregada por género, mayoritariamente se está avanzando hacia eso, pero no hay desagregación por otros determinantes de marginación, como tienen que ver el uso del idioma, la ubicación geográfica, cuán remota esta es, grupos indígenas y otros determinantes de inequidad.
Ada Ocampo [00:13:16] Pero también hay retos políticos, y yo quisiera referirme también a ellos. Que un país tome la decisión política de evaluar los ODS, quizás es el reto más grande. Porque no hay que esperar que lo tome, hay que generar una serie de acciones con diferentes actores para influir en el gobierno, concientizar, levantar conciencia sobre la importancia de evaluar los ODS. Porque es importante tanto para rendir cuentas como para mejores políticas públicas, programas y proyectos, y más equitativos dentro del país. Yo creo que esto es muy importante.
Nataly Salas [00:14:05] Claro, además de los retos que encontramos normalmente cuando hacemos evaluaciones, este tipo de evaluaciones conlleva retos un poco más de mayor peso, por la especificidad de la Agenda 2030, pero también las implicaciones o más bien lo que requieren los países para poder hacer este tipo de evaluaciones. Y en ese sentido, quería preguntarle a Ericka, ¿cómo hicieron en Costa Rica? ¿Cómo empezaron esta evaluación? ¿Qué les motivó a hacer una evaluación con esta especificidad? Y también, ¿qué compromisos asumió el Ministerio? Para poder hacer el eventual uso de los resultados de la evaluación y llevar a la práctica estos beneficios, y solventar los retos que hemos venido conversando hasta ahora.
Ericka Valerio [00:14:59] Primero, aclarar que no hemos realizado una evaluación específica de los ODS. Sin embargo, la Evaluación de la Cooperación tiene un componente importante de análisis de los ODS. Esta evaluación nace de la necesidad de información que se tenía desde la rectoría de la Cooperación Internacional en el país, que es parte de Mideplan. Como primer paso, lo que hicimos fue que se incluye esta evaluación en la Agenda Nacional de Evaluaciones, lo cual es muy importante ya que genera un respaldo político institucional para realizarla, tal como comentaba Ada. Quizá lo más difícil para nosotros fue definir el objeto de evaluación, esto por la cantidad de actores y la información que teníamos disponible. Además, no se había realizado una evaluación temática, esta es la principal diferencia con las otras evaluaciones que tenemos en la ANE, que son evaluaciones programáticas. Esta evaluación tuvo el desafío de encontrar el equilibrio entre las necesidades planteadas y lo que podía hacerse con esta información, con el tiempo y también con los recursos disponibles.
Ericka Valerio [00:16:11] Fue una evaluación más compleja y tuvo que realizarse una contratación previa, esto con el apoyo de FOCELAC en la parte financiera, para poder definir lo que era el alcance de esta evaluación. Se conformaron dos equipos de trabajo. El primer equipo fue un equipo técnico conformado por la Unidad de Evaluación y el Área de Cooperación Internacional, y también de FOCELAC. Y un equipo técnico ampliado que estaba conformado por diferentes dependencias del MINAE. El hecho de tener tantos actores y por tanto tiempo, fue un desafío que implicó un proceso de consulta con todas las partes involucradas. También algo que ayudó mucho en este proceso, sin duda, fue contar con instrumentos metodológicos generados por la Unidad de Evaluación de Mideplan. Nosotros tenemos una serie de guías y manuales, los cuales pudimos adaptar al contexto de una evaluación no tradicional. Y agregar también, que fue muy importante el interés político que se mantuvo durante todo el proceso de evaluación, desde la etapa de diseño, y lo tenemos ahorita actualmente también en la etapa de uso de la evaluación.
Nataly Salas [00:17:23] Ericka, cuando mencionas MINAE, el MINAE es el Ministerio de Ambiente de Costa Rica, ¿cierto?
Ericka Valerio [00:17:28] Sí, es el Ministerio Ambiente y Energía.
Nataly Salas [00:17:30] Sí. Recordemos que las temáticas de la evaluación con componente ODS que hizo Costa Rica eran biodiversidad y cambio climático. Victoria, usted como coordinadora del equipo de evaluación, cuéntenos su experiencia.
Victoria Sánchez [00:17:46] Sí, en general. Fue una experiencia muy enriquecedora, la verdad, porque efectivamente tenía algunos retos particulares. Si no recuerdo mal, la última entrevista que tuvimos antes de la selección del equipo evaluador fue en los primeros días de marzo, justo antes de que nos fuéramos todos a encerrar a casa, y entonces toda la evaluación hubo que hacerla a distancia, cuando en realidad se había concebido para que se pudiera llevar a cabo de manera presencial. Dificultades hubo, claro, muchas, pero tengo que reconocer que creo que por ambas partes hubo flexibilidad como para poder llegar a puntos de acuerdo, y que pudiera avanzar la evaluación.
Victoria Sánchez [00:18:31] Y a mí me gustaría aquí también hacer una reflexión breve sobre en qué medida la demanda de evaluación con la que te encuentras, a la que tienes que responder cuando comienzas a trabajar en la ejecución de una evaluación, hasta qué punto te predetermina. Y claro, si el propósito de la evaluación no tiene nada que ver con los ODS, y si las preguntas de evaluación no se lo plantean, entonces es mucho más difícil. En cambio, si están reflejadas, claro, ya tienes como el mandato para trabajar en esa dirección. Yo diría que siempre que la intervención tenga volumen y densidad suficiente, como para que pueda ser suficientemente relevante el introducir esta mirada, y por otra parte, si la entidad que está encargando la evaluación tiene un interés real, genuino podemos decir, en la temática de ODS, si no hay interés real no se va a hacer nada con esos resultados, no se van a aplicar y no se va a llegar a nada ni a ninguna mejora en este sentido.
Victoria Sánchez [00:19:32] Pero bueno, no fue el caso. En esta ocasión sí que había una pregunta específica que interrogaba sobre eso. En este caso, sobre todo trabajábamos desde un punto de vista de alineación. Lo que teníamos que hacer era analizar todo el conjunto de intervenciones que se habían llevado a cabo, promovidas desde el sector público eso sí, en el ámbito de la biodiversidad y cambio climático, y que habían recibido financiación externa o que había sido Cooperación Triangular o Sur-Sur. Y ahí sí que se pudo desarrollar esa parte del trabajo con la información de la que se disponía.
Nataly Salas [00:20:02] Gracias, Victoria. Pensando un poco en el tema de este episodio del «podcast», que la idea es hablar más del diseño, del alcance de las evaluaciones, quería pedirle a Ada que nos cuente un poco de la guía para alinear las necesidades, las prioridades nacionales con la Agenda 2030, de la que es coautora. Y que nos cuente qué dice esa guía, qué recomienda para diseñar una evaluación, para definir al alcance de una evaluación de ODS o con componente ODS.
Ada Ocampo [00:20:37] Efectivamente, la guía es un esfuerzo conjunto realizado por el Ministerio de Relaciones Exteriores de Finlandia, el IIED que es un «think tank» con sede en Londres, UNICEF, EvalSDGs, y DEval que es el Departamento de Evaluación de la Cooperación Alemán. Fue un trabajo bastante interesante, se inicia con un retiro en Finlandia en el año 2018-19, donde participan una serie de actores entre gobiernos, parlamentarios, representantes de las VOPEs o de las llamadas redes y asociaciones de evaluación, entre la Academia y otros. Participan una serie de actores, para elaborar una guía que defina los pasos que se requieren para hacer una evaluación de los ODS. Esta guía está disponible al momento en tres idiomas, inglés, francés y español, y puede encontrarse en diversos sitios web, siendo quizás el más fácil de mencionar el evalsdgs.org.
Ada Ocampo [00:21:58] La guía toma los diferentes pasos que son necesarios para definir una evaluación de los ODS, desde la ubicación de los actores que van a participar. Ahí es importante mencionar que uno de los principales actores, porque según la Agenda 2030 y desde la Declaración de París de 2005, los gobiernos tienen que estar al timón, uno de los principales actores es el Gobierno, y no podemos pensar que ellos de facto están interesados o tienen el conocimiento suficiente como para impulsar esta evaluación. Hay que dialogar, el diálogo político es importante y más aún en el periodo de pandemia en el que estamos, en donde hay una serie de actores que se preguntan si es que la evaluación es una prioridad al momento. Yo me atrevo a decir que, más que nunca, es una súper prioridad, porque si no conocemos, no generamos evidencia de cómo estamos avanzando, cómo podemos hablar de salir de esta pandemia, de esta crisis humanitaria en la que estamos viviendo, si es que no estamos informados por evidencia.
Ada Ocampo [00:23:14] También habla de otros pasos, como asegurar que la evaluación sea participativa y para esto propone al menos tres instancias. Un comité ejecutivo formado por representantes de varios ministerios y departamentos de Estado, que van a ser los que tienen la capacidad de decisión, un grupo de referencia formado por representantes de la sociedad civil y el sector privado, que va a presentar las perspectivas desde diferentes puntos de vista, y un grupo de expertos o un comité técnico, que va a ser más una caja de resonancia para la calidad de la evaluación. También habla de la fase de recolección de información, poniendo un énfasis importante tanto en la información cualitativa y cuantitativa, y la importancia de revisar o de reconstruir una teoría de cambio. La importancia de permear todas las fases con criterios de equidad y de igualdad de género, y la importancia de triangular la información.
Nataly Salas [00:24:28] Ada, disculpa que te interrumpa. Una de las cosas que me llama la atención que introduce la evaluación de los ODS, es la posibilidad de usar los principios de la Agenda 2030 como criterios de evaluación. ¿Qué diferencia hace esto con las evaluaciones que normalmente hacemos, en las que usamos los criterios del CAD? ¿Qué diferencia hace usar los principios de la agenda en este tipo de evaluaciones?
Ada Ocampo [00:24:54] Verdad que uno de los valores agregados de la guía, tiene varios, pero yo creo que uno de los principales valores agregados de la guía es esa discusión. Realmente lo que propone la guía es usar los principios de la OCDE de CAD, pero adicionalmente, analizar la posibilidad de usar los principios de la agenda, que nadie se quede atrás, los principios de responsabilidad compartida, como criterios adicionales dependiendo del alcance de evaluación, o si es que esto no es posible, que los criterios se tomen como referencia para formular las preguntas de evaluación. ¿Por qué esto es importante? Es importante porque así vamos a garantizar que la evidencia que se genere, esté alineada con lo que la agenda propone.
Ada Ocampo [00:25:49] Por ejemplo, que realmente se esté avanzando hacia el progreso a los ODS con equidad, que se esté avanzando sin dejar a nadie atrás. Pero también es importante mencionar que al incorporar estos principios, estamos también acercándonos per se a algunos ODS que son de por sí transversales, como el ODS cinco que habla de la igualdad de género y el ODS 10 que habla de la reducción de disparidades. Entonces, estaríamos también garantizando un enfoque transversal de los ODS transversales.
Nataly Salas [00:26:32] Muchas gracias, Ada, me queda muy claro. Sin embargo, me surgen más dudas. En términos metodológicos, ¿cómo concretamos esto? ¿Cómo diseñamos la metodología de una evaluación de ODS con estas consideraciones? Y ahí tal vez Victoria pueda contarnos un poco de su experiencia de tantos años haciendo evaluaciones. ¿Cómo verías o qué considerarías fundamental a tomar en cuenta durante la definición del alcance de una evaluación, liderada por un país en temas de Agenda 2030 ODS?
Victoria Sánchez [00:27:21] Es que yo no sé hasta que punto una evaluación de ODS, metodológicamente, ha de ser muy diferente a otra evaluación que no sea de ODS, justamente en el apartado metodológico por decir así. Otra cuestión efectivamente es el alcance y me ha hecho gracia que Ada recordara la Declaración de París, la del 2005, porque yo recuerdo alguna experiencia que se llevó a cabo, en la que se utilizaban los criterios de la Declaración de París como criterios de evaluación, y la verdad es que conseguía obtener resultados bastante interesantes. A mí me parece muy buena idea lo de mezclar los principios de la Agenda 2030 con los criterios de evaluación de la OCDE. Veo el riesgo que a veces nos encontramos quienes hacemos la parte final, quienes hacemos la evaluación, que a veces el alcance es amplísimo, y luego realmente necesitas mucho tiempo y muchos recursos para poder responder satisfactoriamente a todo ello.
Victoria Sánchez [00:28:21] Desde un punto de vista metodológico, y si haces una buena evaluación en general y aplicas eso a una evaluación de ODS, puede seguir siendo una buena evaluación. Es verdad que la guía que mencionaba hace un momento, no hace grandes recomendaciones metodológicas, porque lo que dice es que cada caso será diferente y yo creo que eso además también es un acierto. Curiosamente, por ejemplo, en la evaluación de Costa Rica, metodológicamente sí que se recogían casi todas las recomendaciones que se hacen ahí, que son recomendaciones más de carácter general. Se buscó toda la participación que fue posible en el marco, como ya decía, de la pandemia. Por supuesto, se incorporó varios enfoques transversales, pero muy particularmente el de género, por supuesto se trianguló. Pero todas estas cuestiones deberían estar bastante presentes en casi todas las evaluaciones, para obtener resultados de calidad.
Victoria Sánchez [00:29:12] Entonces, si lo pienso desde otra perspectiva, y pienso bueno, en esta ocasión, ahora, si tuviera que empezar con esta evaluación, qué es lo que haría de manera diferente, pensando sobre todo en los ODS. Y creo que quizá lo que faltó por parte del equipo evaluador, o sea por nuestra parte, fue hablar más con el equipo gestor, el equipo técnico que se llamaba en el marco de esta evaluación, hablar más sobre qué querían conseguir justamente en el ámbito de los ODS. Y esto no es porque no habláramos, porque Ericka puede confirmar que hablamos repetidamente a lo largo de todo el proceso de evaluación, pero quizá específicamente no abordamos tanto este tema de los ODS. Pero ya digo, yo creo que metodológicamente podemos considerar que fue bastante satisfactorio, así que quizá lo que cambiaría sería eso. Y espero haberte respondido, Nataly.
Nataly Salas [00:30:03] Sí, Victoria, muchísimas gracias. Quisiera también pedirle a Ada que nos cuente un poco su experiencia participando en evaluaciones en otros países, porque ella creo que lo mencionó anteriormente, estuvo acompañando el diseño de la evaluación de Nigeria y conoce la experiencia en Finlandia. Entonces, un poco para poner en perspectiva la experiencia que nos está comentando Victoria, con la experiencia que tuvo Ada en estos otros dos países. Si nos cuentas, Ada, ¿cómo fue?
Ada Ocampo [00:30:34] Bueno, yo quisiera empezar por la evaluación de Finlandia, que es una evaluación que tomó como unidad de análisis el Plan Nacional de Desarrollo Sostenible que tiene Finlandia. Y que de hecho, presenta una alineación con la Agenda 2030, aunque no totalmente directa, la evaluación también fue una oportunidad para que una de las preguntas sea en la medida en que este plan está alineado con la Agenda 2030. A nivel metodológico fue interesante, pero yo quisiera referirme un poquito a cómo ellos concibieron la evaluación, para que la evaluación realmente tenga un efecto importante a nivel nacional. Ellos decidieron hacer la evaluación de tal forma que les estuviera lista en un momento de campaña electoral. La evaluación fue comisionada por el primer ministro cuando obtuvo la aprobación del Congreso.
Ada Ocampo [00:31:39] Fue planificada de tal suerte que se terminara en un momento de campaña electoral y se hizo mediante un proceso de difusión amplísimo. Eso hizo que los partidos tomaran las recomendaciones y los hallazgos de la evaluación en su campaña, para decir «Nosotros vamos a incorporar la Agenda 2030. Nosotros vamos a hacer una ruta crítica para asegurar que Finlandia va a conseguir los Objetivos de Desarrollo Sostenible». Y se generó una competencia bastante saludable a nivel de partidos, y el partido que ganó efectivamente tuvo que presentar un plan de cómo haría esto. La metodología previó diferentes instancias de participación, planificadas e inducidas, para generar una mayor apropiación de la evaluación.
Ada Ocampo [00:32:31] Para lograr la participación se generaron instancias de comité ejecutivo, instancias de decisión a nivel de Finlandia y grupos de referencia en diferentes niveles. Entre ellos estuvo EvalSDGs, donde yo estaba participando como copresidenta en ese momento. Fue interesante porque el equipo evaluador hizo consultas sistemáticas, a sí mismo los gestores de la evaluación. Se usaron métodos cuantitativos y cualitativos, y algunos de ellos fueron innovadores para el caso específico de Finlandia, que tenía como uno de sus componentes también evaluar, al ser un país del norte, su política exterior de apoyo a países en desarrollo. Finlandia tomó un camino diferente que ubicó los ODS, después de hacer un rankeo de aquellos ODS que cumplían ciertos requisitos, como por ejemplo, tener evaluaciones previas, contar con apoyo político de parte del sector y algunos otros, contar con presupuesto, ser una prioridad nacional, etcétera. Así identifica sus ODS.
Ada Ocampo [00:33:50] Si bien usan métodos cuantitativos y cualitativos, y se basa en los principios de la agenda para asegurar un enfoque de equidad, de igualdad de género y de sostenibilidad medioambiental, entre otros, también es importante mencionar que hay algunas innovaciones que también hace Finlandia, que también hace Nigeria, que son importantes. Por ejemplo, Nigeria ha trabajado en otras evaluaciones también y también en estas, con un método que se llama confrontar hipótesis rivales. Entonces, hay una hipótesis A respecto a cómo alcanzar los ODS y hay una hipótesis B de cuál sería otra estrategia, que es contrastada. Y a través de la evaluación y a través de diferentes análisis, se contrastan estas hipótesis rivales, dando resultados bastante interesantes.
Ada Ocampo [00:34:49] Nigeria hizo una compilación de varias evaluaciones que se habían hecho antes de la agenda y también durante la agenda, pero que no necesariamente estaban alineadas. Y realizó una metaevaluación, un metaanálisis de estas evaluaciones para determinar varias cosas, determinar el nivel de progreso hacia ODS, aunque no se haya llamado así, y el nivel de alineación. Y utilizó esta metaevaluación con varios propósitos, entre ellos dos a resaltar, utilizó la evaluación para hacer su Informe Voluntario Nacional que se presentó en el Foro Político de Alto Nivel en 2020, y también para informar las evaluaciones de los ODS tres y ODS cuatro, que acaban de concluir y ser lanzados.
Ericka Valerio [00:35:44] Nataly, aprovechando lo que nos cuenta Ada, me gustaría contar algunos aprendizajes que tuvimos también en el caso de Costa Rica, en este proceso de evaluación. Por ser este un proceso diferente, FOCELAC propone realizar una sistematización para identificar los aprendizajes, por ejemplo, la conformación de los equipos interinstitucionales que hicimos. Como les mencionaba anteriormente, esto fue una buena práctica. Sin embargo, al contar con tantos actores y ser este un proceso tan extenso, es importante recordar los roles de los mismos, que fue una de las experiencias que tuvimos.
Ericka Valerio [00:36:28] Otro aprendizaje es sobre la definición del alcance, que bien lo mencionaba Victoria, definitivamente tiene que ser muy acotado, considerando el tiempo, los recursos disponibles y también los actores que van a participar. Y también otro aprendizaje, la importancia de contar con una unidad de evaluación que coordine técnicamente el proceso, y contar con un instrumental metodológico que guíe un proceso riguroso y de calidad, que en este caso la unidad de evaluación no lo tenía. Y bueno, como otro aprendizaje, fue la importancia del apoyo técnico y financiero que recibimos del proyecto FOCELAC.
Nataly Salas [00:37:11] Muchas gracias, Ericka y Ada y Victoria también, por supuesto, por sus aportes. Bueno, creo que el principal aprendizaje que me llevo de esta conversación es que hay muchísimas maneras de hacer evaluaciones de ODS, ahí podemos aplicar las metodologías tradicionales que se han venido usando a lo largo de los años en evaluación, pero también metodologías novedosas. Sin embargo, lo fundamental de una evaluación de ODS es que se puede usar para generar cambio en la política pública de un país, y que permita orientarla o alinearla hacia el cumplimiento de estos Objetivos de Desarrollo Sostenible. Quisiera pedirles un mensaje de cierre, una ronda final, si están incluso de acuerdo con eso que estoy diciendo o no. Pero para ir cerrando esta conversación y que puedan generar una reflexión breve, cada una de ustedes. Victoria, adelante.
Victoria Sánchez [00:38:24] A ver, Nataly, me parece que lo más importante es pensar que si finalmente cuando llegue 2030 y termine 2030, si se quiere conocer no solamente hasta dónde se ha llegado en términos de cuál ha sido el nivel de logro, sino también algo que yo creo que es muy importante, que es el cómo se ha llegado hasta allí. Yo creo que para que se conozca eso hay que empezar a hacer evaluaciones ya, aunque sean parciales. Y aunque sean parciales, pueden generar estos aprendizajes, que bien por vías como la que mencionaba Ada de la metaevaluación, o bien mediante procesos de síntesis o por otros en los que se pueda pensar, se puede ir construyendo todo este aprendizaje y este conocimiento, en relación con no solamente con cuánto se progresa, sino con el cómo se progresa. Y en consecuencia, yo creo que lo que hay que hacer es efectivamente seguir animando a los países y a las instituciones, para que realicen evaluaciones de ODS.
Nataly Salas [00:39:23] Ericka.
Ericka Valerio [00:39:23] Bueno, tal vez mi mensaje sería que en Costa Rica, a pesar de contar con un departamento de evaluación con recursos limitados, tanto en la parte de personas solo somos seis funcionarios, y también en recursos financieros, hemos realizado un esfuerzo por dar los primeros pasos en los procesos de evaluación de los ODS. No ha sido fácil, hemos encontrado resistencia de los actores, sistemas de información incompletos, poca disponibilidad de información, y bueno, ahora el tema de la pandemia también. Pero hemos procurado mantenernos en esta línea, ya que sabemos que la evaluación nos da la evidencia, para que las instituciones del gobierno puedan alinear sus intervenciones hacia la consecución de los compromisos internacionales. Entonces, sabemos que tenemos aún mucho por mejorar, pero sin duda es un inicio que puede motivar a otros países a realizar evaluaciones de los ODS.
Nataly Salas [00:40:22] Adelante, Ada.
Ada Ocampo [00:40:23] Yo quisiera concluir diciendo de que todos estamos por contribuir a un mundo mejor, a un mundo más justo, a un mundo más equitativo. En ese marco, yo creo que la evaluación de los ODS nos brinda la oportunidad, una oportunidad de oro, de ver si realmente estamos contribuyendo o cómo podemos contribuir mejor, el cómo podemos contribuir de mejor manera a ese mundo que aspiramos. Las experiencias de Costa Rica, de Finlandia, de Nigeria y también la de Perú, están contribuyendo con muchos países a construir sobre las lecciones y las experiencias ya pasadas. Como mencionaba Ericka, ellos tuvieron el apoyo de FOCELAC. Nigeria tuvo el apoyo de UNICEF, Perú tuvo el apoyo de UNESCO. Los países no están solos, tenemos el compromiso de demostrar que realmente podemos caminar juntos hacia un mundo mejor.
Nataly Salas [00:41:31] Maravilloso cierre. Muchísimas gracias. Muchísimas gracias a las tres. Querida Ericka, Victoria, Ada, por sus aportes, por supuesto por su disposición a participar en este episodio del «podcast» Objetivos de Desarrollo Sostenible, evaluando el progreso para un futuro más brillante, para un futuro mejor. Y por supuesto, por poner en perspectiva la necesidad de avanzar hacia el cumplimiento de la Agenda 2030, que más allá de un compromiso de los países, es la posibilidad de construir mejores condiciones de vida para todas las personas de este planeta. Recuerden que pueden encontrar todos los episodios de este «podcast» en la página web del Instituto Internacional para el Medio Ambiente y el Desarrollo, iied.org.
Y también les cuento, les aviso que el episodio número tres de esta entrega será sobre el uso de las evaluaciones de ODS. Sin embargo, este episodio estará disponible únicamente en inglés. Muchas gracias por su tiempo. Que tengan un buen día. Hasta luego.
Episode 3
Country-led evaluations: revealing the progress of Agenda 2030
The third and final episode focuses on what country-led evaluations can bring to assessing progress towards the SDGs in the face of the COVID-19 pandemic, which diverts the attention of national and sub-national governments.
The podcast features:
- Ana de Oliveira, policy and research officer at Cooperation Canada
- Dorothy Lucks, executive director of SDF Global Pty Ltd
- Petri Uusikylä, co-founder and director at evaluation company Frisky & Anjoy and research director at the University of Vaasa (Complexity Research Group), and
- Kassem El Saddik, co-chair of EvalSDGs and an independent consultant.
Country-led evaluations: revealing the progress of Agenda 2030? full transcript
[Music]
Kassem El Saddik [00.10] Welcome to the third in a series of the podcast on Sustainable Development Goals – Evaluating progress for a brighter future. This is Kassem El Saddik, I am the co-chair of EVALSDGs network, it's a global network of evaluation practitioners around the globe.
We've discussed so far adaptive evaluation when considering global climate risks, and the second episode we covered how to get started in designing, scoping and defining the SDGs evaluation.
In today's session we will be exploring with our guests the use of the country-led evaluation of the SDGs.
And with that I'd like to introduce and welcome three of my guests for today, Dorothy Lucks, Ana de Oliveira and Petri Uusikylä. Dorothy?
Dorothy Lucks [01.07] Yes, hello to you all. My name is Dorothy Lucks, I'm the executive director of SDF Global, that stands for Sustainable Development Facilitation. I'm a credentialled evaluator and we operate a social enterprise that works not only in evaluation, but also in a whole range of other sustainable development support processes.
Kassem El Saddik [01.34] And good to mention that Dorothy was the former co-chair and one of the founders of EVALSDGs network. Welcome, Dorothy.
My second guest is Ana de Oliveira, Ana?
Ana de Oliveira [01.45] Hello everyone, so Ana de Oliveira, I am the policy and research officer at Corporation Canada, which is a Canadian network focusing on international cooperation.
So I lead the files around the 2030 Agenda and the SDGs within the organisation, and for two years in partnership with 12 other global organisations I coordinated the independent civil society-led report on the progress of national SDGs implementation.
Kassem El Saddik [02.18] Welcome, Ana. My third guest is Petri Uusikylä. Petri?
Petri Uusikylä [02.24] Oh, hello. My name is Petri Uusikylä, I'm a managing director of Frisky & Anjoy, a research company doing evaluations. But I also have a position at the University of Vaasa, I'm a research director in a research platform called the Complexity Research. And that is a systems approach to more complicated and complex societal issues.
But I've just started evaluation of the national SDG goals in Finland, that is a national evaluation. So I'm willing to share also some of the ideas that we have in mind doing the national evaluation. Thank you.
Kassem El Saddik [03.09] Thank you Petri, Ana and Dorothy.
And I think, Petri, you've laid the ground for probably the first set of questions and kicking off the dialogue. And basically I'd like to start with you so we’re clear and you have some clarity on what is country-led evaluation since you've just embarking into it. What is country-led evaluation, why is it conducted and how is it done?
Petri Uusikylä [03.32] That is an important question since Finland has been very active in evaluating the national implementation of the SDG goals. We carried out the national evaluation in 2019, and that was an excellent evaluation, it gave a very good, state-of-the-art analysis where we were at that time.
And also I was very happy to see that quite a few of those recommendations in that evaluations you could see in our government, the forthcoming government programme and its implementation guidelines. So it really fed to the political process as well.
And then Finland has also carried out the so-called voluntary national reviews, and the latest one was in 2020, and that's also an excellent report. And there was Switzerland and Mozambique that gave the peer review comments to that report. And it's an excellent piece of work also.
But why Finland is doing another evaluation at this time? I'm not fully aware of all the motivations behind it, but I think one of the reasons is that SDG goals have been very high in our government agenda. And also quite a few of our ministries want to see the continuation since we have a government election coming in next spring, and that is a way to give the state-of-the-art analysis and results prior to government programme preparation. So I think that's one of the key reasons. And the main emphasis in our evaluation is the implementation, the governance and the mainstreaming issues from the governance perspective. That's one of the main topics.
Kassem El Saddik [05.27] Thank you, Petri. I think, so, mainstreaming issues, governance issues that seems to be carried over from one evaluation to the other within the context of the agenda.
Dorothy, what if it's not a fully-fledged evaluation of the SDGs? What of its only contribution to the evaluation of the SDGs, I mean, from your perspective, what can you bring to the picture and what all the different stakeholders would be contributing to?
Dorothy Lucks [05.55] Yes, thank you, Kassem. There's no doubt that Finland has been a world leader with regard to evaluation of the SDGs and, Petri, we watch Finland with great interest on how a more systematic approach across the SDGs is actually carried out. But most countries find that difficult to achieve, that very often the issues that they're facing are very broad and vast, and therefore the resources to lead a full-scale evaluation of the SDGs at the national level is beyond their reach at this stage.
But what has been happening is where there's a lot of focus on partnership evaluations. So they may not be fully country-initiated and led, but there is strong partnership on particular SDGs. If we take the example of Nigeria, where they went through a process of prioritising which SDGs they would focus on, or in other countries where there's been a focus on SDG two, SDG one, that particular organisations have worked with national agencies and other stakeholders to focus on particular SDGs, and accelerate and deepen understanding of progress around those. And a contribution is still progress and better than nothing at all. And there are useful lessons coming out of these.
So the principles embodied in what Finland are doing, even if they can't be fully replicated, there's a lot that still can be done to push forward on country-led evaluations related to the SDGs.
Kassem El Saddik [07.57] Brilliant, thank you so much. So in a sense, country-led evaluation, while it has [08:04] that revolves around national ownership, whenever capacity and resources are not there we call upon our partners and we capitalise on the partnerships to get it running, to get it done.
And, I mean, from there I'd like to pick up on that discussion with Ana and see and explore with her that although it happens at the country level, but some of the results of those evaluations of the SDGs are picked up also and used at different levels, at various levels.
From your experience, Ana, what other levels do these country-led evaluations, what are they used there?
Ana de Oliveira [08.44] Sure, so these evaluations, they can be used at the global level. So, for example, to measure progress strategically across all countries, as well as national and subnational levels.
So these evaluations at the country level, and particularly the voluntary national reviews – or the VNRs, as we call them – they bring contributions in different areas. So, for example, they outline which governance arrangements exist, which institutional mechanisms are in place to realise the agenda and the SDGs, which policies are being implemented, how multiple stakeholders and this, including non-state actors are being engaged, or how the country's pursuing partnerships. So all these elements can contribute to global level conversations.
And picking up on what Dorothy just said about partnership evaluation, there is an interesting example from the 2021 VNR report that relate to coordination and partnership in reporting. So there was a coordinated effort between six small island developing states – or the SIDs, as we call them – in which they collectively highlighted common vulnerabilities, challenges and strategies, which could then be turned into shared solutions and new common opportunities to contribute to sustainable development initiatives through inter-and intra-regional cooperation. So all these would fit into another level of conversations in the regional and in the global spheres.
Kassem El Saddik [10.47] Of course the country-led evaluation and the VNR, the relationship between them is more or less complimentary in the sense that VNRs might feed into the country-led evaluations and vice versa.
Probably I want to catch up with Dorothy on the use of those country-led evaluations and VNRs globally, both in terms of accountability and in terms of learning. From your perspective, what lessons have we learned so far of the seven years of engaging on the Agenda 2030 with whatever we have in terms of evaluation and VNRs?
Dorothy Lucks [11.21] There are positive lessons and negative lessons, I think that what we've learned over the last years of implementation, that there's no doubt that there has been increase in what we call meta-evaluations, where there's an approach, particularly by some of the development partners, to look across different countries and look at the emerging lessons related to the SDGs or around particular themes which are important across the SDGs. So, say, for instance, meta-evaluations on capacity development or on learning, methods of learning.
So these meta-evaluations contribute very much to the development body of knowledge, and these are used by development partners. We know not only within a specific development partners, but there is quite a lot of collaboration across different networks, like the UN Evaluation Group, like EvalPartners. So those global platforms allow for sharing of information.
Similarly, the VNRs that UN DESA (United Nations Department of Economic and Social Affairs) does generate a synthesis across the VNRs. And this is a very useful resource to look at what aspects are moving well and which need further focus.
But at the same time, there are some lessons around the challenges with that. One is that there are insufficient evaluations actually being carried out, that there has been a strong push to develop national evaluation systems, but perhaps with limited success or a very varied success. This means that the number of evaluations being generated are below what might be possible, below potential. Also, there's not always a systematic approach to looking at country-led evaluations, not just those related to a particular organisation.
Kassem El Saddik [13.35] Thank you, Dorothy, I think you've raised a couple of very interesting points that would probably… we can carry it with Petri. How the evaluation in Finland potentially is informing policy development at the national level? If you give us some examples probably to pick up and then share the learnings with others.
Petri Uusikylä [13.57] There are actually two very important platforms in Finland. One being the Strategic Research Council under the Academy of Finland, that is more, let's say, research oriented, but also has certain evaluative missions included. And that is to - the Strategic Research Council aims to identify the critical phenomena at the very broad level, and that also links to the SDG goals and targets. There are big research programmes that also contain elements of evaluation. So that's the long-term strategic research and evaluation.
And then there is a new instrument called the ENTS, which is the new funding instrument located under the prime minister's office. And that's an annual… they carry out several annual evaluations that are targeted to critical phenomena related directly to the government programme and its implementation.
So when the finance comes from the prime minister's office, then also it's easier to feed the results and recommendations back to policy process because it's located at the high level of government. So that's one of the reasons.
And the second is also that the various ministries are very keen on financing their own evaluations somehow related to the various goals and targets of the SDGs. So that also feeds to the sectoral processes at the ministry level.
So I would say these three are important platforms for doing evaluations.
Kassem El Saddik [15.48] If you allow me on picking up on this, I mean, obviously it seems that there is a systematic approach in conducting the evaluation in Finland, and there is more of a well-crafted process that feed into the policy and making the development agenda. How inclusive are those evaluations? What about the current one, how inclusive it is? And what type of stakeholders and partners are engaged?
Petri Uusikylä [16.19] Well, I have to say that, you know, the VNR in 2020 was extremely inclusive. And when we started our evaluation mission, we went through the VNR and its main recommendations, I was very happy to realise that the involvement of civil society organisations, various NGOs, but also the enterprises and the business community was included.
And the way how they included those stakeholders was kind of applying, I would call it, the contribution approach, and NGOs and companies reported how they have contributed to the certain goals and targets of the Agenda 2030. And that, I think, it's a very important approach.
What we are doing, we also want to involve the various stakeholders and societal groups in our evaluation. Some of the issues that we are evaluating are more administrative, let's say, how the SDGs have been combined with, let's say, performance budgeting or performance management processes. Of course that's the administrative issue. But in other areas we also carry out several interviews and we organise a number of stakeholder work groups where we want to hear the opinions of various parties. So it's included.
Kassem El Saddik [17.53] Thank you so much. Brilliant.
If I can turn the discussion a bit at the subnational level and with other subnational and national stakeholders. So Ana, building on what Dorothy brought up in terms of the global scene, and capitalising on what Petri just said in terms of the use at the national level, what about the subnational ones and subnational actors? What can you bring to the discussion?
Ana de Oliveira [18.25] Yeah, for sure. So I guess we should first recognise that there should be an understanding that all the different levels, say, the local, the regional, the national and the global, they are all intertwined. And so overlooking the local realities would ultimately mean losing important perspectives, experiences and drivers of local action to achieve the SDGs.
And as an example, their colleagues at the United Cities and Local Governments – which is an umbrella organisation focusing on the local level – they have been analysing and reporting on local efforts, meaning efforts driven by cities and by territories, to achieve the SDGs. They have been doing this for years now.
And a lot of learning comes from subnational and local processes. And as you've mentioned, Kassem, we see that more countries are mentioning the voluntary local reviews and the national reviews. So they're mentioning VLRs in the VNR reports.
So this is a very positive trend. It directly relates to SDG 11 on sustainable cities and communities, but this is also positive in the sense that the VLRs work as an instrument of local efforts, and as such they pave the way for subnational accountability structures.
Some examples from 2021 VNRs include Colombia, that published a guide to voluntary local reports for the cities and municipalities of Colombia. Another example is Japan, that established the VLR Lab, which is a platform that collects worldwide information on VLRs.
So for sure this topic is really picking up momentum in several places in the world.
Kassem El Saddik [20.36] Thank you, Ana. I think you've raised a couple of interesting examples of local and some national initiatives feeding into country-led evaluation of the SDGs.
It would be wise to a bit reflect on the main big lessons at the different levels. So probably if we can pick up on a couple of lessons from the global level, Dorothy, if you started there? And then rolling it out at the national and subnational levels, Petri and Ana.
Dorothy Lucks [21.09] Yes, indeed. I think the role of the development partners, particularly when they work together, can be very powerful because there is a level of resource there, which if it's combined with a country-led approach, that actually helps to move a common agenda forward.
So if we take a few examples, for instance, of the Asian Development Bank that's just completed an evaluation of its approach to climate change. This was a very large evaluation that engaged a number of different countries, and particularly the Pacific region, as a region and looked at countries there which are really heavily impacted by climate change.
So in relation to SDG 13, even though the evaluation was really looking at the Asian Development Bank’s approach, it brought together a whole range of different countries to talk about and reflect on their own progress. So there was bringing together country evaluations that had been done, project evaluations, all looking at this approach to climate change.
One important lesson from that was that the decision-makers in the Asian Development Bank and in countries who were engaged right at the start of that evaluation to really look at the key evaluation questions that wanted to be answered to then feed into decision-making. And that has been successful in that ADB has now intensified its approach and is working across different countries to work with them on a range of priorities.
The other good example here is the MOPAN network [Multilateral Organisation Performance Assessment Network], which brings together a whole range of bilateral development partners who contribute to a central fund to do evaluations, not just of their assessments really of development partners, but they also bring out some of the themes that Ana was talking about.
But I want to raise one other point that isn't always considered at the moment, and that's of the need to involve national evaluators. So people from the particular country of focus and even sometimes subnational if there are different languages, if there are Indigenous Peoples, for instance, within a country or not even within one country but across countries. Those kind of approaches haven't necessarily been picked up enough yet in that link between national evaluations, global evaluations and sub-national or thematic evaluations. The integration of language and culture so that the findings of evaluations can be contextualised to the people that really need the lessons arising from these evaluations.
Kassem El Saddik [24.55] Brilliant, brilliant ideas, Dorothy. And I think you've brought into the picture a couple of very interesting aspects revolving around peer learning among countries within a collective evaluation, engaging decision-makers early into process and even designing and scoping it, doing some evaluations, more thematic evaluation, and most importantly this aspect of localising the evaluation.
And on that point probably I'll pick up on Ana, given bringing the voice of the CSOs [civil society organisations], both in the VNRs and the evaluations. What can you share with us, probably picking up on the localisation of the evaluations in such a context?
Ana de Oliveira [25.40] There are many lessons learned in this sense, as well as recommendations. So I could probably extensively list recommendations around each one of the several topics related to SDGs implementation and evaluation. And what Dorothy has just said is tremendously important.
So a key takeaway would be in the sense of the need of working together with non-state actors. So what you've mentioned Kassem of including CSOs, but also being mindful of Indigenous groups or those groups who are in many different cases overlooked by governments at different levels.
So I would say that there are many actors, civil society organisations, and other experts as well, like the academics, that have been collecting, that have been producing, their own evaluation processes and presenting very comprehensive reports that highly contribute to country-level and global-level review processes.
So with this collaborating with civil society and other groups, and advocating for their reports to be acknowledged and given status at the High-Level Political Forum, for example, would be important in terms of: one, participation and partnership coordination; also in terms of transparency in reporting processes; and third, also in terms of ensuring accountability for progress of the 2030 Agenda and SDGs implementation and evaluation.
Kassem El Saddik [27.39] Thanks, Ana, I think you've shed some light on aspects that are critical for both the VNR process and the country-led evaluations.
And we know that evaluations obviously don't happen in a vacuum, there are politics, definitely, but most importantly also there are risks being… the national, sub-national, as well as global level.
And if I can turn to Petri with a bit of a reflection on how does this current country-led evaluation in Finland is integrating or accounting for those potential risks. Can you shine a bit of light on those aspects, Petri?
Petri Uusikylä [28.17] Going to the risks and, you know, the future challenges, I think our National Commission on Sustainable Development is doing an excellent job in coordinating all the activities. And of course, these are very, you know, complicated and complex issues where you face immediately the question of, you know, the systemic impact on certain things. And that is something that goes beyond traditional policy coherence thinking, because there, if you really want to understand the systems change in the field of Agenda 2030 or SDGs, you also have to see more strongly how these various goals and targets, you know, are interconnected, how they influence one another. And you cannot do it if you tackle them separately.
And we have been doing an excellent job in dealing with these issues separately. And we have very active ministries who have done a wonderful job. But when it comes to interdependencies, and systems change, and systemic approach, there we have still much to do.
I think I have to mention that the SDG evaluations as a whole have, you know, created much closer cooperation between ministries – especially the Ministry for Foreign Affairs and other line ministries because the Ministry for Foreign Affairs has a very strong evaluation unit. And now the SDGs are connecting the various line ministries, like the Ministry of the Environment, the ministries of employment and economy and so on and so on, together with the Ministry for Foreign Affairs, which also gives a more global approach to our national implementation.
I think it's a good thing. Whether it's intended or unintended doesn't matter, but you know, it can be seen as a positive outcome of a process.
Kassem El Saddik [30.24] Indeed, often unintended consequences or outcomes turns out to have much more impact on the change that we all aspire for.
Probably on that point, on the point of risks and the era we're living in, we've all noticed that there has been a slowdown in terms of the progress on the Agenda 2030. And I'm afraid that that this slowdown basically is affecting also the uptake and the scale of evaluations of the Agenda 2030.
Quick remarks probably or quick reflections, Dorothy, if you can shed some light here?
Dorothy Lucks [31.04] In terms of the slowdown of the 2030 Agenda, this indeed is a big issue, and one that evaluation as a sector needs to consider and move towards. Evaluations take too long to organise and present at the moment. But there are ways around that, that most evaluations that we're doing at the moment have some element of consideration of this slowdown, they have some consideration of the impact of COVID.
So these… what we need is more dynamic evaluations, which cut across and draw out some of these risks and the responses so that the learnings can be generated quickly and fed back into programming.
Kassem El Saddik [31.56] Of course, thank you so much. I mean, that's critically important, bringing the aspect of dynamic evaluation.
I think we're coming to an end now and we want to probably conclude our podcast. Can I ask you please to formulate and give us at least one thought for our audience on the value of those country-led evaluations to inform the stakeholders about the progress on the Agenda 2030? One key takeaway message from each one of you before we conclude. Let's start with Ana, then Petri and then Dorothy.
Ana de Oliveira [32.35] I just wanted to quickly circle back to what you have said in terms of challenges. And you mentioned a couple of them beginning with C – like COVID-19, climate change, conflicts. And I just wanted to add another one, which is civic space and how it has been shrinking in many parts of the world, particularly after the pandemic and how many countries started using the pandemic as an excuse to further close civic space.
So it would be worth checking a document from the OECD DAC, which is the Development Assistance Committee, a document they launched in July last year, which is the committee's recommendation on enabling civil society. And so this is a common standard for official actors to address civic space and work with civil society towards the 2030 Agenda.
So a key recommendation in this sense would really be for countries to not only adhere to the recommendation, but also embed it into their practices.
And one big concern is that a worrying number of countries is falling short in achieving the global commitments. And even to adhering to the UN Secretary General's guidelines for evaluation and reporting.
So we have eight years left until 2030, so governments and stakeholders from the global community, they should redouble their efforts to address gaps and work towards achieving the SDGs.
So I would leave as a final note that there is opportunity for this to further strengthen evaluation and reporting. And the way of doing that is making use of comparative analysis from using data from all their reports including civil society reports and including peer countries reports. So keeping the dialogue open is a major key takeaway.
Kassem El Saddik [34.57] Brilliant, brilliant idea. Petri?
Petri Uusikylä [35.01] I think it's extremely important to carry out country-led evaluations, the national evaluations, because what we really need is evidence for our policymaking. And I'm not referring to a very, you know, narrow, evidence-based policymaking, I'm referring to the more evidence-informed policymaking where evaluation is an extremely important tool for giving decision-makers updated information where we are in the certain fields of Agenda 2030. That's the first thing.
But second thing is that evaluations, if they are carried out in the open and participatory way, they are also excellent tools for involving new groups and stakeholders into process. And we saw that in our previous national evaluation carried out in 2019. And one of the key factors was also the participatory approach applied in that evaluation. So I think that's a very important issue.
And maybe the third one is that when you carry out national evaluations, I think it's important to apply somehow the developmental evaluation approach, meaning that you are not only collecting data and then reporting at the end of the process, but it's a dialogue also between the commissioner of the evaluation and the evaluation team and the stakeholders. So you feed the process while you are doing the evaluation.
I think that's the lesson learned from Finland.
Kassem El Saddik [36.44] Thank you so much, Petri. I think you've shed the light on very important aspects, of course.
Dorothy, I think if you can conclude with one or two takeaways that you'd like to highlight?
Dorothy Lucks [36.55] Yes, I'd like to build on what Petri mentioned there about the importance of inclusion and participation – that if we're talking about country-led, who's actually doing the leading? Are we talking about country leaders? Are we talking about only one sector? I think we're not, we're actually talking about a broad-based country-led approach where there's a reflection of the diversity in that particular country, or even in that subnational area.
So while country-led evaluations of the SDGs are very large and difficult to achieve, there are digestible chunks that we can move on with very quickly, which is the value of country-led inclusion in evaluations for the SDGs – so that we can have something that's very manageable of changing the way we do practice of any evaluation to make sure that we are touching base with the key sectors of society within that country, those who are impacted by evaluations, making sure that they are contributing to the evaluations in a way that brings value and brings learning on progress for the SDGs and identifies innovations and accelerates progress.
Kassem El Saddik [38.30] Thank you, brilliant ideas.
I would like to thank you so much for your deep reflections and your takeaway messages. And I want to, before saying goodbye to our audience, I'd like to call upon them to sign up for the links and sign up for the different episodes that we've created under this series.
And to also announce that this network and other partners, including EvalPartners, moving towards an initiatives to reinstate government commitments to the evaluation of the Agenda 2030 under a new resolution that promotes country-led evaluation, and discussion is still there.
And I believe what we've covered so far feeds and comes timely into that dialogue and those efforts.
So, thank you so much. And a final call for our audience to read, check and visit the websites of IIED, DEval, and EVALSDGs, and other social media platform, and share widely those episodes. And hopefully we can talk and we can initiate new dialogues on new issues around evaluation of the Agenda 2030.
Thank you and goodbye.
Listen to our Make Change Happen podcast, which provides informal insights into IIED’s work to create positive change and make the complex issues we face more accessible to wider audiences.