Best Practices in Monitoring Indicators: Standardization of Methodologies

By Anne Andayi Misiko

Editors: Maureen Onyango and Linda Davis, Ph. D

Executive Summary

There is need for accurate calculation and reporting of impact indicators by sector players in building the evidence on the economic and environmental benefits of integrating women across the energy access value chain.

Since 2013, the Partnership on Women’s Entrepreneurship in Renewables (wPOWER) Hub has been aggregating data from partner organizations working with women entrepreneurs within communities in India and Africa. These partners promote adoption of clean cooking, heating and lighting solutions at the household level. wPOWER considered a set of standard indicator guidelines from the Global Climate Change (GCC) indicator handbook to assess its collective impact. The GCC indicators are recommended for use as guidelines for impact reporting by those working within the climate change mitigation sector.

In the process of ascertaining data accuracy from its partner organizations, the wPOWER Hub, recognized discrepancies in the adoption and execution of the GCC guidelines. For example, different partners using GCC guidelines reported impact on the same indicator but relied on different methodologies to calculate these indicators resulting in significant discrepancies. In addition, as the wPOWER partnership expanded over several years, more differences were realized among partners who were using self-developed methodologies not in line with stipulated GCC guidelines and further worsening the reliability of impact reported.

This publication provides an overview of the variances observed and provides a recommendation on specific methodologies to be adopted for two key indicators used in measuring the impact of clean energy adoption i.e., amount of greenhouse gas (GHG) (metric tons of CO2), and the number of people with increased access to clean energy and proposes harmonized methods for measuring impact to ensure consistency, legitimacy and accuracy of impact reporting.

Introduction

Due to the long-term nature and diversity of climate change mitigation initiatives, measuring outputs and outcomes of these initiatives can be challenging. Generally, organizations within the climate change mitigation sector rely on a set of standard indicator guidelines from the Global Climate Change (GCC) indicator handbook in reporting impact. GCC guidelines are designed for four climate change program areas including adaptation, clean energy, modern energy sources and lastly sustainable landscapes. Using these guidelines, sector players select best-fit indicators that match their specific project objectives.

This publication focuses on sector players working in clean energy adoption at the household level specifically clean cooking and lighting. Globally, it is estimated that 2.7 billion people continue to rely primarily on traditional biomass fuels such as wood, dung and crop residue for cooking and heating (with detrimental effects to health and the environment), and 1.3 billion people do not have access to electricity at all. Many of the organizations have adopted the GCC indicators as a guide to validate their impact increasing the adoption of cleaner cooking, lighting and heating methods, However, the methodologies applied by the organizations can vary widely posing challenges in aggregation of data to quantify the impact of these clean energy initiatives.

The wPOWER Hub, as a secretariat working with over 40 partner organizations promoting the role of women across the clean energy value chain, aggregates partner data. In the process of aggregating partner data, wPOWER discovered two key challenges. Firstly, some partners were using self-developed (non – GCC) impact indicators and methodologies to assess progress of their projects. Secondly, that even though some partner organizations reported their impact as per the set GCC guidelines, the internal methodologies adopted by each of the partners in calculating impact results were different resulting in large variations in the reported impact results. Compiling aggregate data based on these diverse reports from partners represented a risk of incorrectly assessing the rate of progress in adoption of clean energy.

The importance of standardized indicators and consistent impact reporting practices

Effective monitoring and evaluation helps to give information on the impact of the projects and highlight lessons learned that can be used to inform future initiatives. Indicators are signposts of change along the path of project implementation and are a valid and reliable way of tracking intended results. Indicators are also crucial in monitoring and evaluation as they describe the degree to which a project has contributed to a solution. While monitoring is the ongoing process by which stakeholders obtain regular feedback on the progress being made towards achieving project goals and objectives, evaluation is the rigorous and independent analysis of completed or ongoing projects to determine the extent to which they are achieving stated objectives.   

According to the United Nations Development Fund, without effective monitoring and evaluation it would be impossible to tell if an organization’s objectives have been met and if strategies deployed had any impact.

The World Bank states that the quality of indicators for measuring impact must reliably measure the success of an intervention. However, even though theoretical guidelines may exist on how to develop indicators, implementers may sometimes be faced with a challenge of how to actually quantify these impact indicators.

In response to these challenges, and those experienced while working with partner organizations, wPOWER championed a standardized methodology and calculation formulae based on the GCC guidelines to be adopted by all its partners to ensure accurate aggregation by wPOWER.

Of the 21 indicators grouped by GCC, wPOWER selected 2 (number 2 and 3 below) and developed additional outcome indicators as best suited to evaluate the progress of wPOWER project;

  1. Number of people receiving training in clean energy;
  2. Greenhouse Gas (GHG) emissions estimated in metric tons of CO2 equivalent, reduced, sequestered, or avoided through clean energy activities;
  3. Number of people with access to clean cooking and lighting;
  4. Number of clean energy enterprises successfully started and operational;
  5. Increase in women’s incomes as a function of economic opportunities.

The proposed standardized methodology of calculation was based on two indicators namely; Greenhouse Gas (GHG) emissions estimated in metric tons of CO2 and Number of people with access to clean cooking and lighting.

Calculating greenhouses gas emission reductions

This indicator reports the quantity of greenhouse gas (GHG) emissions, which include carbon dioxide (CO2), methane (CH4), nitrous oxide (N2O) and other gases. The requirement for this indicator is a calculated estimate of metric tons of CO2 reduced, sequestered, and/or avoided. The amount of greenhouse gas emissions is assumed to be reduced through the use of clean energy devices like improved cookstoves, fuels and solar-based technologies.

In calculating greenhouse gas emissions, wPOWER realized that several organizations calculated GHG emissions by simply multiplying the number of products sold by the emission factor of each product model (provided by the manufacturer). This approach did not consider the lifespan of the products.

Recommended methodology for calculating the amount of greenhouse gases reduced through clean energy activities (in tons of CO2)

Formula:  Number of cookstoves or solar lamps sold x Emission Factors x Lifespan of Product

Assumptions: Each stove or lamp sold is in use. That each stove or solar lamp has a lifespan that is absolute as some products do not reach maximum expected durability indicated by the manufacturers.

The wPOWER Hub recommends that in order to accurately report the reduction of GHG, the product lifespan should be considered as the products have a limited term of functional use.

Reporting the number of people with increased access to clean cooking and lighting

This indicator reports on the overall number of people engaged through clean energy adoption interventions. In some cases, organizations reported as increased access if a graduate trainee of an entrepreneurship training program engaged in sales activities with a certain number of households. In this scenario, actual product sales are not factored in calculating the impact.

In this example, if an organization trained a total 140 new people and each trainee reached a minimum of 200 households, the number of people reported as having increased access to clean cooking and lighting would have been calculated as simply the number of people trained multiplied by number of households reached multiplied by the average number of people in each household (5 people/household) giving a result of 140,000 people.

In other instances, organizations reported on the same indicator when an actual cookstove and /or solar lamp was sold to a household. In this scenario, if an organization sold 140 cookstoves and 1,000 solar lamps, the number of people reported as having increased access to clean cooking and lighting would have been calculated as; “the total number of products sold” multiplied by “the average number of people in each household” giving a result of 5,700 people.

The challenge here is that impact results are aggregated simply as the number of people with increased access to clean cooking and lighting even though two different calculation methods were used.

The wPOWER Hub recommends that the second method that considered the actual number of products sold is a more accurate measure to determine the number of people with access to clean cooking and lighting.

Recommended methodology for calculating Number of people with increased access to clean cooking and lighting

Formula:  Number of cookstoves sold x Number or people in the household

LIGHTING

Formula:  Number of solar lamps sold x Number or people in the household

Assumptions: Each product sold is in use. Each product is used by one household.

Conclusion

The wPOWER Hub encourages organizations working to increase the adoption of clean cooking and lighting technologies to apply consistent methodologies, assumptions and formulas so that the sector can correctly assess progress of projects and organizations. Further, harmonizing methodologies enhances accuracy and consistency in ensuring legitimacy and accountability to all stakeholders as well as assessing the impact of specific interventions and staff performance.

wPOWER M&E Partner collaborative meeting to harmonize indicators