Guidebook for Monitoring and Evaluating Ecosystem-based Adaptation Interventions

Introduction

Ecosystem-based Adaptation (EbA) aims to reduce negative climate change impacts on people through working with and enhancing nature and ecosystem services. Understanding what EbA interventions are working and why, and more importantly what is not working and why, is essential for learning about best practices and reducing uncertainties about effectiveness and long-term impacts. This is where good monitoring and evaluation (M&E) of EbA interventions is vital.

This Guidebook has been produced by Friends of Ecosystem-based Adaptation (FEBA) partners to provide practitioners and planners with a clear process for developing and operationalizing effective M&E for EbA. It draws from FEBA’s wide network of practice-based experience, and the collaboration and expertise of the FEBA working group on Monitoring & Evaluation of EbA.

The Guidebook provides:

• Key background information on M&E for EbA projects, including descriptions of key concepts, resources, and examples, and specific sections on: defining and contextualising EbA; situating EbA among related approaches; ‘what is M&E and why is it important?’; and, understanding the challenges of monitoring and evaluating EbA.
• An overview of and step by step process for designing and implementing effective M&E for EbA, with specific sections on: (1) Developing a results framework, (2) Defining indicators and setting a baseline, (3) Operationalising monitoring and evaluation system, and (4) Using and communicating the results.

The Guidebook describes key considerations and components for each step of M&E for EbA projects and points to additional tools and methodologies that can be used in specific circumstances. The emphasis is on going beyond simply measuring the activities and outputs of an EbA intervention, to evaluating the results – namely the outcomes and impacts – of those actions. This is essential for understanding whether EbA measures are having the desired effects to increase both human and ecosystems resilience.

Ideally, the Guidebook should be consulted in the early stages of designing an EbA intervention as it will help clarify the logic underlying the measures, including their intended pathway of change. However, it can also be used if the implementation of an EbA intervention has already begun to help make improvements to the original logical framework and M&E system of the intervention, if these exist, or to develop them, if they are not yet in place.

* Download the full text from the right-hand column. A short overview of the briefing note is provided below – see the full text for much more detail. You can read more about the Guidebook and FEBA here.

*The Guidebook is also available in Spanish.

FOUR STEPS for designing and rolling out an M&E process for EbA

This section describes – in brief – four broad steps that any project team of an EbA intervention can follow. Please refer to the full text for much more detail and supporting text and examples. These steps form the foundation for designing and implementing a robust M&E system.

Step 1: Developing a results framework

This step includes the following sections:

• Setting objectives: what is the intervention trying to achieve?
• Using a results framework to clarify the intervention’s pathway for achieving change
• Why you should use a Theory of Change approach for monitoring and evaluating EbA
• When to use a Theory of Change approach
• How to use a Theory of Change approach
• Potential limitations of a Theory of Change approach
• Additional useful resources

As a first step to developing an M&E system for an EbA intervention, you need to establish clear objectives. You can use a results framework to help you understand the steps needed for reaching the objectives of your EbA intervention. A consensus is emerging among M&E experts that the Theory of Change (ToC) approach is one of the most robust results frameworks to be used in the context of adaptation because it is particularly well-suited for the design, monitoring and evaluation of complex, multifaceted and long-term interventions.

While there is no standardised method for developing a ToC, there are some general elements that you should integrate into your ToC process:

1. Identify the intended impact, ensuring it is clearly defined (see Box — 9 — Important terms in relation to ToC and M&E, page 27, for explanations of ‘impact’ and ‘outcome’).
2. Develop a pathway of change by systematically working backwards from the impact, laying out all the necessary steps along the causal pathway and grounding them as concretely as possible in an evidence base.
3. Identify clear indicators and spell out assumptions (this is covered in Step 2, see below).
4. Define and plan activities.
5. Present the ToC as a flow diagram, accompanied by a narrative.
6. Regularly review and modify the ToC to reflect emerging conditions and new knowledge.

See the full text for much more detail. See figure 4 on page 32 for an example of how to illustrate a Theory of Change, highlighting interconnections between short- and longer term outcome.

Step 2: Defining indicators and setting a baseline

This step includes the following sections:

• What are indicators and why do you need them?
• Types of indicators and the importance of focusing on outcomes and impacts
• Choosing the right indicators
• Setting a baseline and establishing targets
• Additional useful resources

Indicators are units of information (about particular objects, conditions, characteristics or behaviour) that can represent (or act as markers for) the broader environmental, socio-economic or climatic situation. Without tracking indicators that can provide you with information about the outcomes (and eventually impacts) of the EbA measures you have implemented, you will not be able to understand if the measures are effective, i.e. if they are delivering adaptation benefits to people, or to identify barriers to their effectiveness. Measuring outcomes and impacts is more challenging than tracking activities and outputs (Figure 5), which is why many EbA interventions to date have mostly monitored process.

There is no single set of universal or standard adaptation indicators.There are various methods for developing indicators. Some general steps to get you started include:

1. Review the intervention’s ToC.
2. Develop a “long list” of potential indicators.
3. Refine your indicator list.

After selecting a final set of indicators, you need to establish a baseline against which change in the indicators can be measured. A baseline is a description of the initial condition/situation before an intervention takes place. Some relevant baseline information may be available from other ongoing initiatives in the project region, or national statistical systems.

In order to better understand to what extent the project is making progress towards reaching its overall objectives, you will also need to establish a set of specific and measurable targets. To do so, think about what the ideal ecological or socio-economic scenario is that you would like your intervention to achieve by a certain point in time.

Refer to the full publications for more details, including on the different type of indicators and a potential compilation of outcome indicators specifically for EbA. See Table 2, on page 40, for example indicators for measuring longer-term adaptation outcomes that can be achieved through EbA, and suggestions on how, where and when to collect data, and immediate outcome indicators that can be used in mid-term evaluations and/or in case the longer-term ‘gold standard’ indicators cannot be tracked due to lack of data, financial resources or time.

Step 3: Operationalising the monitoring and evaluation system

This step includes the following sections:

• Evaluation design options
• Important data considerations
• Putting data collection into action
• Additional useful resources

As part of making your M&E system operational, you need to give some thought to the evaluation design you intend to use. It should allow you to answer the following questions:

• What difference did the intervention make?
• How did it make this difference?
• What other factors were relevant?

To be able to respond to these questions, data collection needs to be designed in a way that will allow you to distinguish what amount of the total change that has taken place can be claimed by your intervention. This is known as ‘attribution’.

The most rigorous option for determining attribution is using an experimental design (e.g. randomised control trial) in which you compare your intervention to, ideally, several control sites where no adaptation measures have been implemented.

Given the complexities involved in determining exact attribution, you can also assess the contribution of your intervention to achieving adaptation outcomes. To do so, you should consider the above questions and use an evaluation framework that illustrates the contributing factors and the relationships between them (e.g. based on a Theory of Change).

Refer to the full publication for more details, including on data considerations and data collection put into action. See page 20 for information on ‘Understanding the challenges of monitoring and evaluating EbA’.

Step 4: Using and communicating the results

This step includes the following sections:

• Using evaluation results to inform adaptive management
• Communicating to different audiences
• Additional useful resources

Once the monitoring results have been evaluated, you should discuss them with stakeholders involved in (and affected by) the implementation of your intervention (e.g. community representatives, protected area staff, local government, technical advisors), and decide jointly how to adjust interventions and management strategies.

When communicating to different audiences, it is important to:

• Consider the most effective formats for communicating M&E results to different audiences, including written materials, presentations, or more creative outlets.
• Make sure that results are presented in a simple manner and that they are accessible to a wide range of different users.
• Keep in mind the different backgrounds of your audiences (e.g. technical, non-technical, sectoral, cultural, linguistic) and the terminology they are likely to be familiar with.
• Developing an engaging narrative can also help communicate the story of your intervention.

Key external audiences for communicating M&E results include: donors, communities (if not already included during the monitoring), policy makers and wider adaptation community.