Evaluating change can be challenging; it starts with quality data collection
NGOs are increasingly being asked to report on the impact of their work, and good monitoring and evaluation is key. Conservation Through Public Health, a Ugandan NGO, has some lessons to share with other small NGOs and – spoiler alert – it all begins with good data.
Conservation Through Public Health volunteers visit households to deliver health care. They also discuss conservation and the value of Bwindi as a home to endangered mountain gorillas (Photo: Photo Marcel, Culturedfilm)
Three years ago, the organisation that I founded began a journey of reflection to understand the links between the health interventions we support and their conservation outcomes.
Our NGO, Conservation Through Public Health (CTPH), provides local health care services such as hygiene and sanitation advice and access to family planning, and has been doing so around Bwindi Impenetrable National Park (in Uganda) for 12 years. We wanted to know, are health investments paying off for endangered wildlife?
With the help of Oxford University and IIED – and supported by the UK Darwin Initiative – we set out to understand: what is the impact of CTPH’s health interventions? We knew we are positively impacting people’s access to health services, but we didn’t know how or whether this translated into conservation outcomes.
Before we go any further, let me briefly summarise our model: we work with a network of volunteer Village Health Teams (VHTs) to reach out to rural households and deliver health care. The VHTs also deliver something unique: they are trained to be Village Health and Conservation Teams (VHCTs) that sit with households and talk about conservation and the value of Bwindi as a home to endangered mountain gorillas.
In return for their hard work, volunteers receive support to form group income generating projects and are encouraged to reinvest their income into Village Savings and Loan Associations that provide money for volunteers to improve their wellbeing.
So, what did we find?
Our evaluation depended on data collected by the volunteer VHCTs, who are asked to record specific indicators from the homes they visit, measuring their adoption of family planning, good hygiene, sanitation and health seeking behaviour, as well as, good conservation practices.
Unfortunately, as the evaluation team began to dig into the data collected from 2009-17, it became clear that we had issues with data quality. These issues were due to the volunteers having little experience in collecting data, and limited familiarity with the need to be objective and to minimise bias in data collection.
All these issues are completely understandable, but it meant that with poor quality data, not collected in a standardised way, we were unable to easily evaluate the contribution of VHCT’s activities to conservation outcomes.
An M&E guide for non-professionals
While we are sad that we couldn’t assess CTPH’s impact using the data collected by the VHCTs, together we learned some valuable lessons on monitoring and evaluation that we’d like to share with other small conservation and development NGOs internationally.
We’ve created a guide, ‘Monitoring and evaluation for non-professionals: how to ensure quality in data collection processes’. The guide uses our experience in Uganda to describe the steps needed to ensure data collection by volunteers avoids mistakes and unintended biases.
Our guide starts with defining the monitoring and evaluation (M&E) objectives – in our case this is to assess the CTPH’s conservation impact over time, and to implement adaptive management of the VHCT programme. We reflect on creating SMART (specific, measurable, achievable, realistic and timely) criteria, creating simple data collection procedures, and analysing and reporting using the data.
We can all be guilty of thinking about M&E as an add-on, or a really boring chore, but M&E is central to an organisation’s mission. Effective M&E strategies support organisational effectiveness, value for money, better donor and participant relations and inform strategic direction. Thinking about M&E from the start of a project can help to emphasise its core role – so we suggest considering M&E right at the beginning of your project, just after the design.
As we at CTPH look to apply our model elsewhere in Uganda, the Democratic Republic of Congo and other countries in Africa, we’ll be taking our lessons from this project forward.
In the future we expect that we’ll be in a better position to understand the role of the volunteer VHCT programme in contributing to conservation.
We hope you find the guide useful, and we’d love to hear from you – what are your lessons learned from your experience with monitoring and evaluation for small NGOs?
This blog was compiled by Francesca Booker and Gladys Kalema Zikusoka and is based on the hard work of Henry Travers, Alex Ngabirano, Gladys Kalema-Zikusoka, Dilys Roe and E.J. Milner-Gulland.
