Search Results: failure
Learning From Failure 2022
In 2019 and 2020, CARE published Learning from Failures reports to better understand common problems that projects faced during implementation. Deliberately looking for themes in failure has helped CARE as an organization and provides insight on what is improving and what still needs troubleshooting. This report builds on the previous work to show what we most need to address in our programming now.
As always, it is important to note that while each evaluation in this analysis cited specific failures and areas for improvement in the project it reviewed, that does not mean that the projects themselves were failures. Of the 72 evaluations in this analysis, only 2 showed projects that failed to deliver on more than 15% of the project goals. The rest were able to succeed for at least 85% of their commitments. Rather, failures are issues that are within CARE’s control to improve that will improve impact for the people we serve.
To fully improve impact, we must continue to include failures in the conversation. We face a complex future full of barriers and uncertainties. Allowing an open space to discuss challenges or issues across the organization strengthens CARE’s efforts to fight for change. Qualitative analysis provides critical insights that quantitative data does not provide insight into the stories behind these challenges to better understand how we can develop solutions.
CARE reviewed a total of 72 evaluations from 65 projects, with 44 final reports published between February 2020 and September 2021 and 28 midterm reports published between March 2018 and October 2020. Seven projects had both midterm and final evaluations at the time of this analysis. For ease of analysis, as in previous years, failures were grouped into 11 categories (see Annex A, the Failures Codebook for details).
Results
The most common failures in this year’s report are:
• Understanding context—both in the design phase of a project and refining the understanding of context and changing circumstances throughout the whole life of a project, rather than a concentrated analysis phase that is separate from project implementation. For example, an agriculture project that built it’s activities assuming that all farmers would have regular internet access, only to find that fewer than 10% of project participants had smartphones and that the network in the area is unreliable, has to significantly redesign both activities and budgets.
• Sustainability—projects often faced challenges with sustainability, particularly in planning exit strategies. Importantly, one of the core issues with sustainability is involving the right partners at the right time. 47% of projects that struggled with sustainability also had failures in partnership. For example, a project that assumed governments would take over training for project participants once the project closed, but that failed to include handover activities with the government at the local level, found that activities and impacts are not set up to be sustainable.
• Partnerships—strengthening partnerships at all levels, from government stakeholders to community members and building appropriate feedback and consultation mechanisms, is the third most common weakness across projects. For example, a project that did not include local private sector actors in its gender equality trainings and assumes that the private sector would automatically serve women farmers, found that women were not getting services or impact at the right level.
Another core finding is that failures at the design phase can be very hard to correct. While projects improve significantly between midterm and endline, this is not always possible. There are particular kinds of failure that are difficult to overcome over time. Major budget shortfalls, a MEAL plan that does not provide quality baseline data, and insufficient investments in understanding context over the entire life of a project are less likely to improve over time than partnerships and overall MEAL processes.
Some areas also showed marked improvements after significant investments. Monitoring, Evaluation, Accountability, and Learning (MEAL), Gender, Human Resources, and Budget Management are all categories that show improvements over the three rounds of learning from failures analysis. This reflects CARE’s core investments in those areas over the last 4 years, partly based on the findings and recommendations from previous Learning From Failure reports. Specifically, this round of data demonstrates that the organization is addressing gender-related issues. Not only are there fewer failures related to gender overall, the difference between midterm and final evaluations in gender displays how effective these methods are in decreasing the incidence of “failures” related to engaging women and girls and looking at structural factors that limit participation in activities.
Another key finding from this year’s analysis is that projects are improving over time. For the first time, this analysis reviewed mid-term reports in an effort to understand failures early enough in the process to adjust projects. Projects report much higher rates of failure at midterm than they do at final evaluation. In the projects where we compared midline to endline results within the same project, a significant number of failures that appeared in the mid-term evaluation were resolved by the end of the project. On average, mid-term evaluations reflect failures in 50% of possible categories, and final evaluations show failures in 38% of possible options. Partnerships (especially around engaging communities themselves), key inputs, scale planning and MEAL are all areas that show marked improvement over the life of the project.
Read More...
As always, it is important to note that while each evaluation in this analysis cited specific failures and areas for improvement in the project it reviewed, that does not mean that the projects themselves were failures. Of the 72 evaluations in this analysis, only 2 showed projects that failed to deliver on more than 15% of the project goals. The rest were able to succeed for at least 85% of their commitments. Rather, failures are issues that are within CARE’s control to improve that will improve impact for the people we serve.
To fully improve impact, we must continue to include failures in the conversation. We face a complex future full of barriers and uncertainties. Allowing an open space to discuss challenges or issues across the organization strengthens CARE’s efforts to fight for change. Qualitative analysis provides critical insights that quantitative data does not provide insight into the stories behind these challenges to better understand how we can develop solutions.
CARE reviewed a total of 72 evaluations from 65 projects, with 44 final reports published between February 2020 and September 2021 and 28 midterm reports published between March 2018 and October 2020. Seven projects had both midterm and final evaluations at the time of this analysis. For ease of analysis, as in previous years, failures were grouped into 11 categories (see Annex A, the Failures Codebook for details).
Results
The most common failures in this year’s report are:
• Understanding context—both in the design phase of a project and refining the understanding of context and changing circumstances throughout the whole life of a project, rather than a concentrated analysis phase that is separate from project implementation. For example, an agriculture project that built it’s activities assuming that all farmers would have regular internet access, only to find that fewer than 10% of project participants had smartphones and that the network in the area is unreliable, has to significantly redesign both activities and budgets.
• Sustainability—projects often faced challenges with sustainability, particularly in planning exit strategies. Importantly, one of the core issues with sustainability is involving the right partners at the right time. 47% of projects that struggled with sustainability also had failures in partnership. For example, a project that assumed governments would take over training for project participants once the project closed, but that failed to include handover activities with the government at the local level, found that activities and impacts are not set up to be sustainable.
• Partnerships—strengthening partnerships at all levels, from government stakeholders to community members and building appropriate feedback and consultation mechanisms, is the third most common weakness across projects. For example, a project that did not include local private sector actors in its gender equality trainings and assumes that the private sector would automatically serve women farmers, found that women were not getting services or impact at the right level.
Another core finding is that failures at the design phase can be very hard to correct. While projects improve significantly between midterm and endline, this is not always possible. There are particular kinds of failure that are difficult to overcome over time. Major budget shortfalls, a MEAL plan that does not provide quality baseline data, and insufficient investments in understanding context over the entire life of a project are less likely to improve over time than partnerships and overall MEAL processes.
Some areas also showed marked improvements after significant investments. Monitoring, Evaluation, Accountability, and Learning (MEAL), Gender, Human Resources, and Budget Management are all categories that show improvements over the three rounds of learning from failures analysis. This reflects CARE’s core investments in those areas over the last 4 years, partly based on the findings and recommendations from previous Learning From Failure reports. Specifically, this round of data demonstrates that the organization is addressing gender-related issues. Not only are there fewer failures related to gender overall, the difference between midterm and final evaluations in gender displays how effective these methods are in decreasing the incidence of “failures” related to engaging women and girls and looking at structural factors that limit participation in activities.
Another key finding from this year’s analysis is that projects are improving over time. For the first time, this analysis reviewed mid-term reports in an effort to understand failures early enough in the process to adjust projects. Projects report much higher rates of failure at midterm than they do at final evaluation. In the projects where we compared midline to endline results within the same project, a significant number of failures that appeared in the mid-term evaluation were resolved by the end of the project. On average, mid-term evaluations reflect failures in 50% of possible categories, and final evaluations show failures in 38% of possible options. Partnerships (especially around engaging communities themselves), key inputs, scale planning and MEAL are all areas that show marked improvement over the life of the project.
Read More...
Learning from Failure 2020
Part of striving for the deepest and most sustainable impact at the biggest scale possible is understanding what doesn’t work. CARE’s commitment not only to the highest quality programming, but also to continual improvement, drives us to celebrate our successes and to examine our failures. In 2019, CARE published our first Learning From Failure report, where we looked at what project evaluations told us was going wrong, and areas where we can strengthen our programming to improve our impact. By analyzing broader trends across several projects CARE can get a broader sense of systemic weaknesses that lead to failures in specific cases. We pair this with our podcast with individual case studies where we look at specific examples of failures and how to address them so we can illustrate trends with illustrative examples. That gives us the space to make bigger strategic changes to address underlying causes of failure and support teams to improve work at all levels. One example of this is targeting CARE’s investments in Monitoring Evaluation, Accountability, and Learning (MEAL) systems and capacity building to address common failures we found. In 2020, we repeated the analysis to see where we are improving, and where we still need work. Read More...
Mawe Tatu II
Mawe Tatu is a Swahili concept that translates to three pebbles. This name was chosen to illustrate the physics of dropping three pebbles into a pond to represent interventions related to women, men and youth as main components of society. As the pebbles fall, they have the immediate effect of moving the water. Each pebble also creates ripples that propagate and interact with those of other pebbles, in space and time. It is a metaphor that expresses, respectively, the short- and medium-term effects of the project as well as the long-term impact that will result in synergistic effects of Mawe Tatu's interventions. Like Mawe Tatu I, the programme works with women and youth and men to improve the socio-economic status of women and youth and their powers to influence decisions at the household and community levels.
The overall objective of this evaluation is to measure the progress and results of the programme based on the Theory of Change, to draw lessons (lessons learned) for future programmes.
To achieve this, the evaluation team used the mixed methodological approach (quantitative and qualitative) to collect data from programme participants, key informants and Mawe Tatu II programme documentation. A range of techniques were associated with this approach including: the document review to understand the logic of the Programme, focus group discussions with the participants of the Programme in the 8 health zones of the evaluated programme (Rutshuru excluded because of the context of the war between the M23 and the FARDC), a household survey which facilitated reaching several households in 9 health zones as well as 2 health zones and two control health zones. In addition, semi-structured interviews with key informants were conducted enriched by direct observations to identify different attitudes related to the evaluation questions.
Overall, this evaluation indicates that the Mawe Tatu II programme has had significant successes in terms of its 3 trajectories linked to the first Outcome, related to access (and control) to capital, knowledge, skills and entrepreneurial mindset, and the commitment of men, and other successes at the level of its other two trajectories (market access and sexual and reproductive health) as we have also noticed in the database of harvested Outcomes shared with us in the Mawe Tatu II programme documentation at the beginning of this final evaluation.
Programme participants testify that thanks to the training and awareness-raising activities in the VSLAs to which they have joined, they have made savings through which they access capital to launch their small to large businesses. Some women and young members of the few VSLAs have accessed loans in some MFIs although for MFIs and banks in general, the services do not yet offer financial services adapted to women and young entrepreneurs who are members of VSLAs and remain indifferent to cooperate and carry out business with VSLAs because they continue to find them less reliable because they lack collateral to give to taking credit. At the same time, mistrust of formal financial institutions, caused by the failures that entrepreneurs have witnessed in the past, has also created a barrier, which was not fully addressed during the programme, in the effective use of services and products. Read More...
The overall objective of this evaluation is to measure the progress and results of the programme based on the Theory of Change, to draw lessons (lessons learned) for future programmes.
To achieve this, the evaluation team used the mixed methodological approach (quantitative and qualitative) to collect data from programme participants, key informants and Mawe Tatu II programme documentation. A range of techniques were associated with this approach including: the document review to understand the logic of the Programme, focus group discussions with the participants of the Programme in the 8 health zones of the evaluated programme (Rutshuru excluded because of the context of the war between the M23 and the FARDC), a household survey which facilitated reaching several households in 9 health zones as well as 2 health zones and two control health zones. In addition, semi-structured interviews with key informants were conducted enriched by direct observations to identify different attitudes related to the evaluation questions.
Overall, this evaluation indicates that the Mawe Tatu II programme has had significant successes in terms of its 3 trajectories linked to the first Outcome, related to access (and control) to capital, knowledge, skills and entrepreneurial mindset, and the commitment of men, and other successes at the level of its other two trajectories (market access and sexual and reproductive health) as we have also noticed in the database of harvested Outcomes shared with us in the Mawe Tatu II programme documentation at the beginning of this final evaluation.
Programme participants testify that thanks to the training and awareness-raising activities in the VSLAs to which they have joined, they have made savings through which they access capital to launch their small to large businesses. Some women and young members of the few VSLAs have accessed loans in some MFIs although for MFIs and banks in general, the services do not yet offer financial services adapted to women and young entrepreneurs who are members of VSLAs and remain indifferent to cooperate and carry out business with VSLAs because they continue to find them less reliable because they lack collateral to give to taking credit. At the same time, mistrust of formal financial institutions, caused by the failures that entrepreneurs have witnessed in the past, has also created a barrier, which was not fully addressed during the programme, in the effective use of services and products. Read More...
Ecological Assessment and Detailed Scoping Study in the Sundarbans and Hakaluki Haor Ecologically Critical Areas (ECA) for Nature-based Solutions (NbS) and Locally Led Adaptation (LLA)
The Ecologically Critical Areas (ECAs) in Bangladesh, notably the Sundarbans and Hakaluki Haor are vital for biodiversity and livelihoods. The UK government’s Adaptation in Ecologically Critical Areas known locally as NABAPALLAB- (Nature Based Adaptation towards Prosperous and Adept Lives & Livelihoods in Bangladesh) project aims to enhance resilience of the ecosystem and livelihoods through Nature-based Solutions (NbS) and Locally Led Adaptation (LLA). The project has conducted three separate assessments (Ecological Assessment and Detailed Scoping study; Stakeholder Mapping and Political Economy Analysis; and Baseline Study) simultaneously during its inception stage. This report synthesizes findings from the ecological assessment and detailed scoping study. Insights were gathered to understand ecological status, community needs, and intervention opportunities in the Sundarbans and Hakaluki Haor. The report further highlights current and emerging threats, drivers and complexities in both ECAs. The status of the natural resource base, degradation pattern and ecosystem health were also analysed and documented using physical data and recent satellite imagery.
In the Sundarbans, the study found different ecosystems, including wetlands, low-lying lands, riverside areas, homesteads, croplands, and coastal embankments. All of these ecosystems are affected by various climatic stressors such as salinity, tidal inundation, erratic rainfall, and cyclones. They are also impacted by anthropogenic drivers such as shrimp farming, water logging, overharvesting, illegal fishing, deforestation, poaching. For example, the study shows that shrimp cultivation has a profound effect on agriculture with changes in land use and intrusion of saline water, which lead to decreased soil fertility, crop failure and lower productivity. Shrimp cultivation also contributes to surface water pollution, leading to scarcity of drinking water and negative impacts on WASH. The study also highlights significant threats to important endangered wildlife species in the Sundarbans ECA, including the Masked Finfoot and White-Rumped Vulture. In addition to the anthropogenic threats, the study has identified other problems such as fishing with poison, bird hunting, use of illegal fishing equipment, canal siltation, and intrusion of saline water into freshwater wetlands. These issues result in decreased fish population, depletion of wildlife and reduced agricultural productivity.
The Hakaluki Haor, encompassing 276 interconnected beels (wetland lakes) over an area of 18,000 hectares, faces significant threats from both anthropogenic and climate change stressors. Extensive sedimentation and agricultural expansion have led to the conversion of over 20 beels to agricultural production, impacting wetland productivity, aquatic vegetation, fish populations, and local livelihoods. Degradation of rivers, canals, and hill streams within the Haor has further exacerbated these problems and challenges, affecting crop production, fish yields, and biodiversity. Encroachment on protected swamp forest areas after phase-out of previous projects, wide-ranging deforestation, and conversion of raised land (Kandas) for agriculture have negatively impacted the habitats of fish and birds, and other wetland-dependent wildlife as well as waterfowl migratory routes. Furthermore, elite capture excludes poor fisher folk and landless individuals from fishing and accessing natural resources, while agricultural expansion in Kanda areas raises concerns about increased use of chemical pesticides and fertilizers that pollute wetlands. Read More...
In the Sundarbans, the study found different ecosystems, including wetlands, low-lying lands, riverside areas, homesteads, croplands, and coastal embankments. All of these ecosystems are affected by various climatic stressors such as salinity, tidal inundation, erratic rainfall, and cyclones. They are also impacted by anthropogenic drivers such as shrimp farming, water logging, overharvesting, illegal fishing, deforestation, poaching. For example, the study shows that shrimp cultivation has a profound effect on agriculture with changes in land use and intrusion of saline water, which lead to decreased soil fertility, crop failure and lower productivity. Shrimp cultivation also contributes to surface water pollution, leading to scarcity of drinking water and negative impacts on WASH. The study also highlights significant threats to important endangered wildlife species in the Sundarbans ECA, including the Masked Finfoot and White-Rumped Vulture. In addition to the anthropogenic threats, the study has identified other problems such as fishing with poison, bird hunting, use of illegal fishing equipment, canal siltation, and intrusion of saline water into freshwater wetlands. These issues result in decreased fish population, depletion of wildlife and reduced agricultural productivity.
The Hakaluki Haor, encompassing 276 interconnected beels (wetland lakes) over an area of 18,000 hectares, faces significant threats from both anthropogenic and climate change stressors. Extensive sedimentation and agricultural expansion have led to the conversion of over 20 beels to agricultural production, impacting wetland productivity, aquatic vegetation, fish populations, and local livelihoods. Degradation of rivers, canals, and hill streams within the Haor has further exacerbated these problems and challenges, affecting crop production, fish yields, and biodiversity. Encroachment on protected swamp forest areas after phase-out of previous projects, wide-ranging deforestation, and conversion of raised land (Kandas) for agriculture have negatively impacted the habitats of fish and birds, and other wetland-dependent wildlife as well as waterfowl migratory routes. Furthermore, elite capture excludes poor fisher folk and landless individuals from fishing and accessing natural resources, while agricultural expansion in Kanda areas raises concerns about increased use of chemical pesticides and fertilizers that pollute wetlands. Read More...
Lesson Learned from the construction of a 1800m3 capacity gabion in Wadi Hassan Valley, Khanfer district, Aden governorate under Food for Assets (FFA) Project
What is the specific situation that the lesson learned relates to?
It is about this asset that serves and protects more than 5,000 acres of agricultural land from drought and adds value in different aspects such as increasing underground water level of Abyan and Aden, as such, leading to diversified livelihood options e.g. livestock rearing and bee farming.
How is this impacted by the local context/environment/culture?
The agricultural sector is one of the most important economic sectors in Abyan governorate, and the main source of income for most of the people, as many of them are engaged in agriculture activities. Abyan governorate is famous for its agricultural valleys including Wadi Banna, Wadi Hassan, Wadi Delta Abyan, Delta Ahour.
Because of previous conflicts and wars that occurred in Abyan, the irrigation system was destroyed and was subjected to destruction and neglect. The Abyan Delta agricultural area located in the districts of Zanzibar and Khanfar in Abyan governorate experienced high flow of water from seasonal rainfall, however, the flow of water irrigated a small part of agricultural areas in Khanfar and Zanzibar districts. The bulk of these flood water went into to the sea, as well as causing damages such as eroding farmers' lands, damaging roads, damaging irrigation channels, bridges, and even the destruction of homes that affected some villages and population centres.
After the failure of the dam project in Wadi Hassan in year 1992, many irrigation channels, including Hussein Canal, were deprived of floodwater, which led to the drought of agricultural lands, in the process, depriving more than 2000 families of their main source of income. Hussein Canal covers more than 5000 Hectares of agricultural land that has been deprived for more than ten years of seasonal floods, which is its main source of irrigation by torrents.
In this project, five villages (Al-Dergag, Al-Komblyah, Maykalan, Kadmat Al-Saeed qasem and Obar Otman) that are inter-connected as a sub-district were targeted and benefited from the floodwater that came through the Hussein Canal. Based on the community leaders and irrigation office’s request, a 1800m3 capacity Gabion (360 inter-connected sub-gabions each with size 5m length X 1m depth X 1m breadth) covering a distance of 105 meters was constructed in Wadi Hassan to bring water from the valley to Hussein main channel for irrigation for villager’s lands by floods and torrents water. [5 Pages]
Read More...
It is about this asset that serves and protects more than 5,000 acres of agricultural land from drought and adds value in different aspects such as increasing underground water level of Abyan and Aden, as such, leading to diversified livelihood options e.g. livestock rearing and bee farming.
How is this impacted by the local context/environment/culture?
The agricultural sector is one of the most important economic sectors in Abyan governorate, and the main source of income for most of the people, as many of them are engaged in agriculture activities. Abyan governorate is famous for its agricultural valleys including Wadi Banna, Wadi Hassan, Wadi Delta Abyan, Delta Ahour.
Because of previous conflicts and wars that occurred in Abyan, the irrigation system was destroyed and was subjected to destruction and neglect. The Abyan Delta agricultural area located in the districts of Zanzibar and Khanfar in Abyan governorate experienced high flow of water from seasonal rainfall, however, the flow of water irrigated a small part of agricultural areas in Khanfar and Zanzibar districts. The bulk of these flood water went into to the sea, as well as causing damages such as eroding farmers' lands, damaging roads, damaging irrigation channels, bridges, and even the destruction of homes that affected some villages and population centres.
After the failure of the dam project in Wadi Hassan in year 1992, many irrigation channels, including Hussein Canal, were deprived of floodwater, which led to the drought of agricultural lands, in the process, depriving more than 2000 families of their main source of income. Hussein Canal covers more than 5000 Hectares of agricultural land that has been deprived for more than ten years of seasonal floods, which is its main source of irrigation by torrents.
In this project, five villages (Al-Dergag, Al-Komblyah, Maykalan, Kadmat Al-Saeed qasem and Obar Otman) that are inter-connected as a sub-district were targeted and benefited from the floodwater that came through the Hussein Canal. Based on the community leaders and irrigation office’s request, a 1800m3 capacity Gabion (360 inter-connected sub-gabions each with size 5m length X 1m depth X 1m breadth) covering a distance of 105 meters was constructed in Wadi Hassan to bring water from the valley to Hussein main channel for irrigation for villager’s lands by floods and torrents water. [5 Pages]
Read More...
Final Evaluation of Climate Change Adaptation of Women Smallholders and Cotton Producers from Vidarbha Region, Maharashtra
CARE India’s project on “Environmentally Sound and Climate Resilient Cotton Production Practices (ESCRCPP)”, supported by Group Galeries Lafayette, was implemented from 2018 to 2021 in 10 villages of Jalgaon Jamod block of Buldhana district in Vidarbha region of Maharashtra, aimed to promote environmentally sound, climate-resilient and inclusive cotton production. The location of the project was guided by the fact that Vidarbha accounts for half of the cotton area of Maharashtra, which has one third of cotton growing area in India and Buldhana district is the major cotton producing area in Vidarbha. The cotton based agricultural economy of the region faces challenges like water stress, climate variability and issues related to farming practices and technology. These result in low productivity, frequent crop failure, poor management of water resources, and high indebtedness of farmers.
The project had three purposes (sub-objectives or outcomes) and six expected results (outputs). The outcomes were: women small holders have the capacity to engage in environmentally sound and climate-resilient cotton production; strengthened collectives for building solidarity, promoting gender equity, and facilitating access of women smallholders from poor and vulnerable households to resilience building resources, services, and opportunities; and enabling environment in the form of supportive cotton value chain actors.
This report is 82 pages long. Read More...
The project had three purposes (sub-objectives or outcomes) and six expected results (outputs). The outcomes were: women small holders have the capacity to engage in environmentally sound and climate-resilient cotton production; strengthened collectives for building solidarity, promoting gender equity, and facilitating access of women smallholders from poor and vulnerable households to resilience building resources, services, and opportunities; and enabling environment in the form of supportive cotton value chain actors.
This report is 82 pages long. Read More...