Adaptive evaluation: considering climate risks in theory and practice full transcript [Music]
Stefano D’Errico [00:00:06] Hello and welcome to the first episode of a short series of podcasts, entitled Sustainable Development Goals: evaluating progress for a brighter future.
My name is Stefano D’Errico and I am head of monitoring, evaluation and learning at the International Institute for Environment and Development, IIED.
[Music]
In this series we'll explore how the national policies and interventions are going to achieve sustainable development. Because sustainable development is every country's goal, but how we get there isn't always straightforward. Too often projects and programmes work in silos and ignore the interactions within, between and around them, and this can hamper progress.
That's where evaluation can play a part. Evaluation of progress against a particular Sustainable Development Goal can identify the interdependencies between the social, economic and environmental dimensions of policies and interventions.
And now we have obviously come out of a key climate moment with COP26 and climate change, which are the most striking example of how action towards one goal can dramatically affect programmes addressing the other Sustainable Development Goals.
So under new and very different climate conditions, the existing adaptation practices and processes will no longer be viable, and progress against other goals will be seriously undermined. Thus, it's vital that climate risks are identified and integrated into evaluations of different thematic areas – education, health, clean water and sanitation, and all the other things that are covered by the SDGs.
So in this first episode we are discussing how evaluators can do that. We'll hear what Kenya is doing, we'll think about the challenges and also the best way to address those challenges – by using a systematic approach.
So without further ado I will ask my guests to introduce themselves.
Samson Machuka [00:02:26] My name is Samson Machuka and I’m currently serving as director of the monitoring and evaluation directorate within the National Treasury and Planning of the Republic of Kenya. And evaluation has been one of our core, main activities, as far as monitoring and evaluation programmes are concerned.
Emilie Beauchamp [00:02:50] And I'm Emilie Beauchamp, I'm a senior researcher at the IIED in the Strategy and Learning Group. My research focuses mostly on the nexus between environments, climate and evaluations. And that means my work involves designing evaluative and research frameworks to assess progress in climate and development policies, but also working with national and local governments on evaluation and monitoring evaluation and learning systems.
Nick Brooks [00:03:20] Hi, I'm Nick Brooks, I'm a researcher and consultant, and my work focuses on climate change adaptation, particularly on climate change risks and how we can integrate considerations of these risks into development, including sustainable development.
I worked extensively on monitoring, evaluation and learning issues around climate adaptation in both an academic and a practitioner context in the space of international development.
And I'm the director of Garama 3C, a small consulting firm based in the UK, and I'm a visiting research fellow at the University of East Anglia.
Stefano D’Errico [00:03:53] So, Nick, I've just talked about climate risk. Perhaps to start us off, you could tell us what we mean by that term?
Nick Brooks [00:04:02] Sure, Stefano. So, there are lots of different types of climate risks, and typically in the field of international development and sustainable development we're thinking about risks associated with particular interventions – development projects, programmes, sustainable development initiatives and so on.
So when we think about projects and programmes, particularly from a monitoring and evaluation perspective, we often think in terms of outputs and outcomes. So on the one hand we can think of risks to a project's outputs and outcomes and ultimately to the impacts that we want to see, but we can also think about risks from a project, from an intervention. So we've got these sort of risks facing in two directions: to the activity we're talking about and from the activity. So I'll say a little bit about those in turn.
So we might have risks to a project or an intervention’s outputs. These, obviously, are things like goods and services we're trying to deliver in the short term. So this might be as simple as climate disasters such as floods or storms, hurricanes, simply destroying infrastructure that's been put in place by a project, or preventing access to project sites, for example.
We might also have risks to these project or programme outcomes. We might be looking at an intervention to enhance agricultural productivity, this could be climate-smart agriculture, sustainable agriculture. Or the objective of an intervention might be, for example, to improve the health of a particular ecosystem or a particular set of natural resources.
So this might all be very well under current conditions, but as a result of climate change we might find that we have things like water shortages, additional pressures on ecosystems and so on. And this might make it more difficult to achieve the outcomes. We might increase sustainable agricultural productivity for a while, but then find that as water resources become more scarce, that isn't sustained and perhaps can't be sustained in the future.
We also have the risk of a project or an intervention being redundant. For example, we might be looking at an intervention to try and sustain resources – natural resources, biological resources, for example – that simply won't be viable under climate change and they'll just disappear. So we have to ask whether trying to sustain something that's ultimately not going to be there in the future is sort of worthwhile. And these are really hard calls to make.
So I'll say something briefly now about the risks from an intervention. We might have an intervention that increases the vulnerability of a particular group of people – a classic example would be something like agricultural expansion that displaces pastoralists or makes it more difficult for them to access pasture during dry seasons, making them more vulnerable to climate change and climate change impacts.
An intervention might also increase environmental vulnerability – again, if we're looking at coastal developments then these might create a sort of hard infrastructure that prevents coastal ecosystems migrating inland. Now, if we're looking at sustainable development interventions, hopefully we wouldn't be in that sort of situation. But that's sort of a pertinent example there.
And finally, if we fail to account for and understand how climate change might change what is viable in the future, we might create entire systems that are based on false premises. We might make an entire economy, or society, or region, dependent on resources that are not going to be there in the future because of climate change.
Stefano D’Errico [00:07:22] And why are climate risks important when it comes to evaluating progress against the Sustainable Development Goals?
Nick Brooks [00:07:30] Yeah, so, this is the critical question, really. I think the bottom line here is that something that is sustainable under current conditions might not be sustainable under future conditions because of climate change.
So if we are ignoring climate change and we're evaluating a sustainable development intervention, and we're saying, ‘Yes, this looks great, it looks like it's going to be economically sustainable, technically, financially sustainable, it looks like it's going to deliver the outcomes that are intended’, and we're not thinking about how climate change might alter the landscape, the context, in which this intervention plays out, then we might just be getting things completely wrong.
So I think the task for sustainable development evaluation is to ensure that sustainable development initiatives, interventions, activities, have taken these risks on board. Because if the evaluation doesn't ask this then it's pretty pointless, particularly in high-risk contexts.
Stefano D’Errico [00:08:28] So what we are saying, really, is that it's vital to include considerations of climate risk in sustainable development evaluation, and in sustainable development goal evaluations.
And, Samson, I know that Kenya is doing this. Can you tell us more about it?
Samson Machuka [00:08:45] Yeah, thank you, Stefano. I think the question you're asking is very pertinent particularly to us as a country. And you asking that question at the time we have had just a big conference in Kenya involving our governors – you know, we have governors who are heading our 47 counties – and the theme of their discussion was mainly climate change and how it poses major risks in terms of our virtual development.
What I need to mention at this stage is that Kenya’s taken climate change issues very, very, very seriously, to the extent that in our Kenya Vision 2030 it is one of the key pillars, key goals, that is being focused on in terms of our planning processes. And therefore climate change issues have been captured at national and county levels in their main five-year manifestos and development plans.
So, what we are doing is they have those programmes and we have come up with critical indicators that have been put in place to ensure that they are being tracked, and therefore to make sure that the climate change issues are addressed effectively in terms of making sure that the green emissions are actually reduced drastically over the five-year plan.
So it is one of our key goals in our development agenda. And I believe that we are headed somewhere, because our indicators are really behaving positively in terms of making sure that there is a lot of responses to the results of the climate change policies that are being implemented.
Stefano D’Errico [00:10:33] And it's great to hear that Kenya is really considering climate risk, and not only in evaluation but also in national policies.
So I really wonder whether it's been always so straightforward. What I'm wondering is what kind of challenges you have encountered?
Samson Machuka [00:10:52] Yeah, the kind of challenges we are facing here are definitely that climate change is not a one-time or an instant thing that can come to be addressed – it’s long term.
So the kind of challenges we are facing is that obviously resources have been a big problem, and also making sure that the policies or the wish of the government to the people still takes time to be cultured – you know, to any culture, the people to start addressing climate change issues is not a very easy issue. But nevertheless it is going on.
And then I can also say that the lack of adequate resources, particularly in terms of making sure that sensitisation programmes are taking place on time, and also making sure that the evaluation issues are strongly mainstreamed in our laws and so forth, I think this is where we have a slow uptake. But nevertheless we are moving in and we hope that that will take shape.
Lastly, maybe something else I can say there is that the other critical challenge which I can also say is that coming up with very, very, very smart indicators that can give us evidence on the basis of how we are doing at climate policies effectively and how they are impacting the whole process, is where we have a slight challenge. Because it's not easy to get the actual data, and it's also not very easy to zero down to that empirical information that can now tell you for sure that this change is purely coming from the implementation of our climate change aspect.
So those are the small challenges that we are facing. But I think with time we will be able to overcome, it takes a little while but that's where we are.
And we are getting the support from international community, we are also allocating submissions before our policy, mandate policy’s already attracting 1% of the budget and once we have those resources available with us then we can now be able to have competent evaluators who can work the ground to make sure that they are getting credible sources of information that can be used to inform policy in terms of how we are moving in changing the climate aspects, so that that we have amicable progress reports.
So credibility of evidence is where we have that kind of challenge – particularly emanating from the climate change – so that we are able to report substantively that this is really where we need change, or this is what we need to change, and influence policy and so forth.
So I think it's a bit of a challenge, but we are hopeful that we are moving headways.
Thank you.
Stefano D’Errico [00:14:00] Great, thanks Samson. It is really interesting to hear, you know, how you're addressing these challenges. And I was wondering how in light of the challenges and of the climate act that has been approved, how do you think evaluation is going to change in Kenya? How do you think evaluation is going to address these issues?
Samson Machuka [00:14:28] That's a good question. We have prevailed since monitoring and evaluation policy has already been approved. The most important thing then is that now evaluation results will be required to inform policy, because in the past policies have not been informed by anything, because policies have been made on an arbitrary basis.
But now with evaluation reports coming out very well out of worksite findings of the evaluators, then it’s expected, it’s actually required, that those policies now, or the reports, must be able to influence and inform policy of what is required to be done. And therefore the evaluations are going to be mainstreamed and be used by the implementers.
But I think for climate change, climate change requires elaborate evaluation so that that kind of evaluation can then be able to influence government budgeting system and policies, so that at least then we put in the implementation mechanisms to ensure that the results or the findings of those evaluations are implemented to the letter. And I think that way we can then be able to say that we are doing something meaningful in terms of climate change development.
Stefano D’Errico [00:16:01] Thank you, Samson.
And, Emilie, we heard from Samson that Kenya has already got that far, is already doing a lot to address climate in policies and in evaluation. Is that the same in all countries?
Emilie Beauchamp [00:16:18] Well, unfortunately not really. There is a spectrum of different experiences and levels of integration of climate in evaluative work. Some countries have started to integrate climate issues in the evaluative frameworks of the country, and some countries are considering climate risk in their thematic evaluations when they're assessing specific development progress.
And what I mean by that is, for example, countries like Costa Rica, Colombia and Mexico have started to monitor and evaluate climate indicators across development, while other countries have successfully done cross-thematic evaluations that look at several areas of sustainable development, including climate as one of the themes.
So climate is increasingly a theme of evaluations and a focus of monitoring and evaluation or M&E systems. Although there is still an overwhelming focus on mitigation or reducing emissions and being climate smart, if you want, rather than also looking at adaptation or how we adapt to ongoing and to future climate risks.
Mostly integrating climate risks into evaluations and recommendation is not happening very often systematically, so you have cases here and there but you don't have a constant practice of doing it – especially not across all countries. And that really involves both countries from the global South and the global North.
So, in general, one of the problems is that – and echoing what Samson has mentioned right before me – there's a lack of resources both in terms of human resources within government evaluation teams, but also a lack of resources in terms of financing to target the themes of climate in evaluation and the integration of climate risk.
There's also a general lack of awareness and commitment at higher political levels that slows things down, really. For example, in most countries, just like Kenya, governments decide which are the priority programmes and policies they want to evaluate on a yearly basis. So evaluation tends to be short-lived, and in that way you have limited information that can be useful to predict future impacts. But also that can lead to having smaller political impact.
I'm saying this is within governments, but it's also true at the international level from, for example, large multilateral development institutions and climate and development funds that are often funders of evaluations. So in that sense evaluations are often used for accountability reasons to justify how resources have been used, rather than truly as a tool for adaptive management and learning.
Stefano D’Errico [00:19:18] So, how can we address the fact that there is different progress across the board? I mean, it sounds like we need to be more systematic, don't we? What would be the first steps towards that, maybe?
Emilie Beauchamp [00:19:32] Yeah, you're right, Stefano, there is a need to be more systematic. And actually the good news is that just like some other aspects of responding to the climate crisis, we do already have the necessary data, sources of knowledge and methods to understand climate risk and shocks.
For example, national meteorological agencies now have good climate data across a lot of countries – most countries – and enough projections. And the quality of Earth observations or satellite geospatial data is increasing every year.
Down to the community level, more and more people are aware and understand the impacts of climates on their communities and ecosystems.
So we do have data and information, but the blockage is changing the traditional evaluation approaches from looking primarily retrospectively at impacts towards a more forward-looking outset.
In fact, evaluations are primarily looking back at what effects an intervention have had in order to recommend improvements in the future. But that premise doesn't really hold because of the uncertainty of climate changes and shocks, as Nick mentioned earlier.
So the value of information of current evaluative approaches is limited, especially if climate risks are not considered. So the really first step for the new generation of evaluation is a need to focus on capturing how climate risks have affected progress, or lack of progress, in development, and then looking forward when doing the evaluation.
So, in a nutshell, evaluators need to change their thinking from what worked to what will work.
Stefano D’Errico [00:21:19] Nick, can you tell us more about forward-looking methods?
Nick Brooks [00:21:24] Sure, I'll have a go.
Yeah, I think you raised a really good point there, Emilie, about the use – we have a lot of data that we can call on now, and we need to start thinking about how we can use that data, that information, about climate risks, about potential future changes and impacts to inform evaluations, but I think maybe more importantly to inform the activities that are being evaluated.
And then the job of the evaluation is to ask how well those activities have actually incorporated that knowledge and that information and those data about climate change and climate change risks.
So that means that this can be pretty challenging for, you know, development and sustainable development undertakings to actually develop a good understanding of climate change risks and to develop good ways of addressing them.
Historically, we've seen probably quite simplistic, deterministic approaches where people look at climate projections, a narrow range of variables, and say, ‘OK, we expect it's going to be on average about three degrees warmer and rainfall is going to decline by 10-20% by this date’.
Now, of course climate change is much more uncertain than that. We're very confident about changes in temperature, I think, certainly at the global level and at the regional level – and maybe more locally, even. When it comes to things like rainfall and more complex hazards it's a lot more difficult to sort of look forward.
So we have these emerging techniques, things like robust decision-making, where you use climate projections to inform your thinking about risk, but you don't let them drive it. You think more about the conditions under which certain activities, processes, behaviours, systems, fail and under which they succeed, and then you think about how likely these might be to be met in the future under climate projections – but also considering unknown unknowns outside of the range of projections.
So, as you can probably tell, this can get very complicated very quickly. So while I think there is a central place for using climate information and climate data in the programming of development actions, and also to consider in evaluations the extent to which this has been done, there are also some other sort of more general things that we can look at as well if we're trying to do more forward-looking evaluations.
There have been a number of sets of principles developed for guiding adaptation and for assessing adaptation. So we might look, for example, to the IIED principles for locally-led adaptation. We also have something that I worked on, again with IIED colleagues, and this is the so-called CAMELS framework. And CAMELS stands for climate adaptation, monitoring, evaluation and learning systems.
So, again, just to summarise that rather convoluted explanation, yes, we can do forward-looking evaluation by either using or assessing the extent to which climate change data projections have been used in planning and project design. But we can also do this forward-looking practice by asking whether the thing that's being evaluated has adhered to certain key principles around, for example, use of science and traditional knowledge, use of appropriate information around transparency, participation, equity, and so on.
Stefano D’Errico [00:24:41] So that is a very good and useful framework for practice, Nick.
And I was wondering what happened next then, Samson, what are the key next steps for Kenya?
Samson Machuka [00:24:54] The next key steps for Kenya is to ensure that the evaluation culture is strongly entrenched in our system. And in fact we already have a very strong organisation in Kenya called the Evaluation Society of Kenya, which is really working closely with our directorate so that we also give evaluators autonomy.
You know, when you want to do an evaluation of government programmes, you cannot do it from [an] internal platform, so you need to give some of independence of people outside government to execute an evaluation because then they are impartial. So we already have the Evaluation Society of Kenya, which has come out very strongly, and it’s really [important ph?] that they would be required to coordinate, of course, in the collaboration with us, the selection of competent evaluators, wherever they are, who can come and do evaluation in specific sectors of the economy – particularly the climate change issues that are now affecting us.
So that's what we are doing, and we believe that we will be able to mobilise resources in the view that we already have a policy in place of 1%. So ministries will have to put 1% of their development budget to support evaluation programmes so that when an evaluation is done then nobody can reject it.
So we are moving towards that direction and therefore evaluation is going to be a key thing. And then we are even proposing for evaluation to take a top-notch position in government. We’re elevating the Department of Monitoring and Evaluation, which is within the National Treasury, to become semi-autonomous. And, you know, when you are semi-autonomous then you can now work independently. Because we know the subject matter is not a popular subject matter. You know, when you talk evaluations not everybody's amused about evaluation. So we want to have some independence. So that's actually [what] we are moving to and I believe we are soon getting there.
Stefano D’Errico [00:27:01] Yeah, thanks Samson.
And Emilie, in terms of supporting evaluators more broadly, what that means in terms of next steps?
Emilie Beauchamp [00:27:09] Yes, well, as Nick mentioned, there are a lot of resources available and data, but a lot of information is still spread across different sectors and sources. So it can be difficult for evaluators to start making sense of applying a new approach or rather a new outlook. That is why IIED will be publishing a guide to support a more systematic approach of how to integrate climate risk into evaluations.
The guide will include an overview of key climate issues to take into account across sustainable development sectors or Sustainable Development Goals, if you want. But it will also include a higher-level approach of how to use principles from the 2030 Agenda for Sustainable Development and filter in climate issues and climate risks.
So this will help evaluators use a new lens in their evaluation. But there will also be a step-by-step approach to help guide evaluators. There will be suggestions of questions to inform the evaluation. For example, we suggest moving from just asking, for example, if there is an evaluation about poverty and food security, ‘has the programme improved the livelihoods in food security of the most marginalised group?’, to additionally asking, ‘have there been climate shocks and changes that have affected how livelihoods and food security improved or did not improve over the course of the intervention?’. But also we need to ask, ‘how will a similar programme need to be adapted to continue being relevant for the future marginalised group in light of future climate risks?’.
And this report is really based on years of shared experience between the authors, new empirical research and collaborations with government evaluators like Samson. We’re expecting the report to be launched shortly in 2022.
Stefano D’Errico [00:29:00] So, with that guide as a support, it sounds as if the process could be made more straightforward. But then what needs to happen to make this systemic change, more training, more money?
Perhaps you’d all like to tell us what you would prioritise and why? Samson first, then Emilie, and finally Nick.
Samson Machuka [00:29:21] Yeah, I think resources are important – you mentioned about resources – resources still remain core if evaluations have to be very, very effective.
And I also need to mention one impediment, that perhaps I needed to have mentioned earlier, is the fact that evaluation must be evidence-based. And evidence means data, getting data that can help you compile an evaluation report.
But the efficacy, the reliability and the timeliness of the data we are talking about may not be very easy to come by, and even the people you want to give you the sources of that information, is an area that requires to be addressed because sources of information, the type of information you are getting, might not be a very easy thing for you to be able to execute a very credible evaluation process.
So to me that's an area that perhaps will also require to be looked at because the source of information is very, very important.
Thank you.
Stefano D’Errico [00:30:29] Thanks, Samson.
Emilie, do you want to say something?
Emilie Beauchamp [00:30:33] Yes. I would very much agree with that, Samson, and it's also good to remember that evaluation needs to be done in collaboration to start creating effective changes. Just like we need more than one source of data and knowledge to do cross-sector evaluations and to address climate issues, what I'd like to see happening more is collaboration and connections between institutions, between government departments and key decision-makers.
Information, but also policymaking, is really siloed at the moment and that prevents cross learning, and that prevents really addressing climate risks. So collaborating between department is the only way to understand the implication of the synergies and the trade-offs between different climate risks and different policy impacts. And also collaboration can help with sharing limited resources.
Stefano D’Errico [00:31:26] Nick?
Nick Brooks [00:31:27] Yeah, so, building on what Emilie and particularly Samson have said, I think this idea of mandating evaluation is really important, and I think Kenya has some good precedent here.
I think the other thing to just reiterate is – and, again, Emilie’s touched on this as well – evaluation processes need to be inclusive and they need to include diverse voices, particularly the voices of the people who are going to be affected by the interventions we're evaluating.
And again this goes back to addressing some of the risks and trying to ensure that that these risks are considered and hopefully avoided, for example, risks of increasing the vulnerability of certain groups, which are related to risks of displacement exclusion, human rights abuses and so on. So, you know, a sort of ‘do no harm’ principle demands that we take an inclusive approach.
But I think more importantly, or at least as importantly, if we have inclusive and diverse voices then we get a better picture of how well things are working on the ground. And that sounds sort of obvious and trite, but I think too often that's not done.
And again in the name of sustainable development, we have activities around things like conservation, carbon offsetting and rewilding, and there will be risks there that are analogous to these risks of increased vulnerability. We hear a lot about people being excluded from areas on which they depend in the name of these sorts of outcomes. So I think again inclusion is really important there.
And finally, I think certainly training is critical. We need people who are designing interventions but also the people who are evaluating those interventions to have a better understanding of climate risks and how consideration of climate risk can be integrated better into sustainable development planning, programming and activities.
I think too much sustainable development and wider development activity fails to grapple with some of the really profound climate risks we're going to be facing, and is quite often limited quite tokenistic sort of incremental adaptation, generalised resilience building, generalised conservation, and what we need to be looking at is how climate change alters the dynamics of sustainable development and how we can ensure that development is sustainable in the face of climate change.
I think just a sort of final word on this from me would be that if development isn't adaptive, if it doesn't incorporate adaptation to climate change, then it's not sustainable. And that's the bottom line. That's what I think evaluation of sustainable development with a climate change lens needs to grapple with.
Stefano D’Errico [00:33:53] Okay, that's great, Nick, a bold statement and an excellent way to end our conversation.
So, thanks to all our guests, Samson Machuka...
Samson Machuka [00:34:03] Thank you.
Stefano D’Errico [00:34:03] Emilie Beauchamp...
Emilie Beauchamp [00:34:04] Thanks, Stefano.
Stefano D’Errico [00:34:05] Nick Brooks...
Nick Brooks [00:34:06] Thank you.
Stefano D’Errico [00:34:07] Please join us for the next episode.
[Break in recording]
Thanks for listening to this episode of Sustainable Development Goals: evaluating progress for a brighter future.
On the next episode we discuss the process of defining the scope and focus of an evaluation – something considered by many to be a major stumbling block to measuring progress against the Sustainable Development Goals.
To find out more about this podcast, our guests and their organisations, visit the IIED website, IIED.org.
See you next time.
Soundcloud 
Stitcher 
TuneIn
Spotify
Listen to to our Make Change Happen podcast, which provides informal insights into IIED’s work to create positive change and make the complex issues we face more accessible to wider audiences.