ISEAL’s thoughts on SOAS systematic review of agricultural certification schemes

Agriculture workers on a strawberry farm © World Bank Photo Collection
Agriculture workers on a strawberry farm © World Bank Photo Collection
SOAS recently published a systematic review of the impacts of certification on farmer and worker well-being in agricultural production.

The resulting report, Effects of certification schemes for agricultural production on socio-economic outcomes in low- and middle-income countries, includes both ISEAL member and non-member standards.

Reviews such as this are essential to understand effectiveness, identify strengths and weaknesses and drive improvements in sustainability standards. ISEAL was a member of the project advisory committee and welcomes the learning this review offers and agrees with many of its recommendations for certification programmes and research methodology insights. 

The review provides mixed results on the impact of certification on key outcomes such as yield, prices, workers’ wages, farmers’ income from certified production and total household income. The rigorous screening process the research team adopted meant that the number of individual studies reviewed for quantitative analysis of these key themes is quite low. For instance, only five studies were reviewed to assess the impact of certification on yield despite there being wider empirical evidence available, which was not included in the review on methodological grounds. This reflects important learning for the research and standards community on both the quality of research studies undertaken on certification and adequate description of study methodologies which was a key component of the selection process.

Improving the quality, diversity and rigour of research on the impact of certification

As the SOAS report notes, researchers and standards have made important advances in robust research to measure the impacts of certification in recent years. However, although there is an increasing body of research and evidence on certification, much of this did not ‘make the cut’ for this review due to insufficient methodological reporting or rigour. The fact that these studies did not meet the review’s inclusion criteria is in itself an important learning for the research and practitioner communities. ISEAL is keen to work with both standards and researchers to disseminate this learning and ensure that future research and reporting is of adequate rigour to be included in such systematic reviews. 

Our experience also suggests that the levels of research rigour often demanded are difficult to achieve in reality – both for researchers and practitioners. There are a number of reasons why it is very difficult to do this kind of counterfactual-based research in the field of sustainability standards and certification, as the nature of the intervention places some limits on what kind of research methodologies are possible. These challenges are well documented and ISEAL is working to improve understanding and innovation in research methodology and design to produce more relevant, representative and rigorous research. We also believe that while all research methods might not be included in a review such as this, there should be space for a variety of methods in this field as they all offer rich insights, often to improve contextual understanding and grounding of interventions.

The SOAS report also highlights that the evidence base on the impacts of certification is limited. The researchers note, and ISEAL agrees, that the evidence base is heavily skewed in favour of one or two certification schemes and almost entirely focused on coffee and cocoa. ISEAL is working with its members and researchers to ensure that over time, this body of research increases. Newer schemes such as the Better Cotton Initiative, Bonsucro and Linking Environment and Farming (LEAF), which were less represented in the SOAS review, are making an important difference in agricultural sustainability but are rarely the focus of external research and evaluation. 

More needs to be done to boost overall farmer household income

Although the review findings indicate that certification increases the price and income farmers receive from the sale of certified produce, it also finds that certification does not seem to improve total household incomes for farmers and wages for workers.

We agree that a more holistic approach is needed to boost farmers’ household incomes as a whole. It is not sufficient for farmers to earn more just from the sale of certified crop, especially if the share of household income from that crop is low. However, raising farmers’ income requires action from more than just sustainability standards. Many factors that help raise farmers’ incomes are often beyond the control of commodity based standards, for example, infrastructure that is needed for marketing and processing requires government intervention.

However, there are a few different ways in which certification schemes are addressing this issue. The first is to boost sales of certified crop to ensure that all available certified production is sold as certified (and does not get disposed as conventional produce). The second is to explore efficiencies and collaborations to reduce the audit cost that farmers might incur to enter and maintain certification or provide financial assistance in meeting these costs. Schemes can also do more to understand the actual costs of certified production and assist farmers in overcoming any barriers, for example, through partnerships with other local development agencies or governments. A third is to support diversification efforts – both in farming and in other income-generating activities so households can diversify their income sources and reduce risks from being overly dependent on one crop.

Beyond these measures, an important step taken by ISEAL and its agricultural members is trying to understand farm economic models better and developing a definition and measurement for what a ‘living income’ for farmers would be from a given model. Building on our work around living wage, there is growing interest among many supply chain actors in understanding whether smallholder farmers are actually earning a ‘living income’ and, if not, what it would take to get them there. The Sustainable Food Lab, GIZ (German Development Agency) and ISEAL are co-hosting a series of events to better explore two connected themes on living income: a common definition and methodology for a living income benchmark and approaches to assessing actual farm and household incomes, and modelling the impact of different interventions.

On workers’ wages, many ISEAL member standards covered in this review are a part of the Global Living Wage Coalition that brings together six of the world's most influential sustainability standards to improve wage levels in certified supply chains. ISEAL members Fairtrade International, Forest Stewardship Council (FSC), GoodWeave, Sustainable Agriculture Network/Rainforest Alliance (SAN/RA), and UTZ, along with Social Accountability International (SAI), have made a joint commitment and launched an unprecedented collaboration that will scale up their impacts to achieve better wages. The group recognises that the vision of a ‘living wage’ for workers is crucial to their individual certification programmes and they have agreed to a shared approach for measuring living wage. The long term goal and shared mission of the Coalition is to see improvements in workers' conditions, including wage levels, in the farms, factories and supply chains participating in certification systems and beyond.

Delivering important outcomes that determine impact

The study concludes that standards do not guarantee impacts, and indeed they cannot, but nor can any other development scheme or sustainability programme.  However, credible standards that are driven by continuous improvement deliver important outcomes that are determinants of impacts.

The review highlights that the impacts of certification are highly context-specific. Context will always have some influence on the impacts that certification may or may not have. This is one of the reasons that credible sustainability standards systems adopt a multi-stakeholder approach to ensure a range of stakeholders are consulted at all stages. In many sectors, ISEAL members are looking at creating change at a sector-wide level or at jurisdictional levels. In agriculture and group certification, value chain dynamics have a significant influence on whether certification has impact or not.

In order to drive change at the scale we seek, it’s fundamental that we continue to work in collaboration forming partnerships with governments, businesses and other actors to improve effectiveness, drive efficiencies and reach scale.

Improving understanding and implementation of rigourous research

ISEAL consistently works with its member standards to improve the understanding of what rigourous research is and how to achieve that rigour in their work. The ISEAL Impacts Code prescribes in Clause 8.6 detailed quality assurance requirements for undergoing outcome and impact evaluations. However, these requirements do not prescribe how the research methodology is reported in studies, which is a key reason for them not being included in the SOAS systematic review. ISEAL will reflect on how its work through its Codes of good practice and with member staff can be geared to improve the quality and reporting of evaluation research in this field. More can be done to improve the quality of impact evaluation research coming both from the standards themselves and the wider research community and we are keen to facilitate cross-learning to keep the methods debates alive.

ISEAL has published a learning paper covering lessons learnt about research design and methods from the baseline phase of three impact evaluations. This will help inform and improve the robustness of evaluation research.

Ultimately, this study provides much learning for sustainability standards and ISEAL will work with its member systems and the SOAS research team to learn from this review.