Rosie Forsyth, ISEAL Community Support Coordinator
In a thought-provoking session at the ISEAL Public Day Conference Dr. Kristin Komives, Monitoring and Evaluation Manager at the ISEAL Alliance, and Dr. Elizabeth Kennedy, Director of Evaluation & Research at the Rainforest Alliance, took a closer look at how to assess the impacts of certification. This was considered primarily through the lens of the ISEAL Code of Good Practice for Assessing the Impacts of Social and Environmental Standards (Impacts Code), recent reviews of impacts literature, and specific case studies.
Impacts Code to Create Feedback Loops
In this session, the main features of the Impacts Code, its implementation, and what is needed in future were explored. The Impacts Code is a credibility tool harnessed by researchers and ISEAL Members that can provide the scope for collaborative initiatives. A core component of the Code’s on-going monitoring and periodic impact assessment comprises of ‘learning feedback loops’. These can be utilised to conceptualise how one intends to effect change, assess to what extent this occurs and then refine work based on such findings. In turn, feedback loops can shape the future direction of research by identifying gaps in research to address.
Current Impacts Evidence is Narrow
In reviewing a variety of literature on the subject, Komives presented the merit of credible counterfactual impact assessment (counterfactuals serve as indications of what outcomes would have been in the absence of intervention. A credible counterfactual works to address selection bias in certification). She cited the need for greater clarity in results, better defined aims, and more meaningful measures of certification impacts assessment.
Drawing on case studies, Komives asserted a growing trend that the current impacts evidence base is narrow and weak: often only a minority proportion of studies contain credible counterfactuals. Many are limited in focus and geographic concentration. The need to critically evaluate the prioritisation of non-certified entities as the most relevant comparison group (counterfactual) as opposed to rural development or conservation interventions was brought into question and sparked debate amongst attendees. Komives presented a variety of issues which merit greater attention. This includes consistency of local application; considering more in-depth themes, such as gender and empowerment impacts, or quality; and achieving broader consensus on impact definitions to allow for more comparable results.
Demonstrating Impact: ‘Why’ over ‘What’
Dr. Kennedy presented an unpacking of certification to measure adoption of specific Best Management Practice (BMP) within a certification system. This considered how to demonstrate impact moving from anecdotal evidence to randomised repeated controls (RRC) and solid empirical evidence. While the storytelling element proves important in the spectrum of understanding, Kennedy stressed the need to move towards an empirical body of research for measuring impacts and understanding results. Furthermore, she highlighted the importance of articulating what certification aims to achieve using measurable indicators of success, stating that researchers must show ‘not just what we do, but why we do what we do.’
Currently, ‘noise’ in results that utilise analyses of certified versus non-certified groups means we need to more closely evaluate the application of BMP and think about other independent variables along the results chain. Results will show the impact of BMP but will not isolate influence of these practices as dependent variables change from economic, social and ecological perspectives. A credible counterfactual, i.e. farms that do not implement BMPs, thus has great value.
Selection Bias Elimination – the Force of a Credible Counterfactual
A common theme was the need to address selection bias in certification, namely the certification of ‘low hanging fruit’ (operations that are already complying with all or most certification criteria) and self-selection bias in getting certified. This can be achieved by using experimental and quasi-experimental design to construct a credible counterfactual, the advantages and challenges of which were also discussed. Rainforest Alliance’s own work offered new approaches for impacts assessment along these lines, such as ‘layering’ by considering one or more specific BMPs across standard systems such as forestry, rather than looking at a set of certified versus non-certified categories.
A potential solution offered was the somewhat controversial issue of creating a randomly divided control sample for studies by adopting an approach used in the pharmaceutical industry to pay participants for the trial or study.
Emerging Themes
Key issues to emerge from the discussion included the need for credible, valid and robust methodologies with regards to validity concerns, selection biases, ethics, and the consistency of applying indicators.
Underlying factors that influence types of research were discussed, such as when studies are donor-driven. The need to allow standard systems to work differently and in different contexts was also expressed. Participants expressed the need to agree on universally applicable indicators in the community. Addressing practicalities such as different desires from the owner of label, producers and consumers was also deemed critical. Taking this forward, Komives advocated BMP as an interesting comparative tool with which more impacts can be delineated.
Absence of Evidence is not Evidence of Absence
Furthermore, it was contended that a well-constructed Theory of Change (ToC), as enshrined in the Impacts Code, should answer both internal and external validity questions. For example, this comprises of adaptive management and evaluative questions along a results chain, such as short versus long term outcomes. Others also mentioned the role common sense has to play in the impacts demonstration continuum on which a ToC is based; a key message being ‘absence of evidence is not evidence of absence’ where more rigorous methodologies are lacking.
Accelerating Community Engagement
A general desire for coordination within the conservation community was expressed. The corporate sector, as well as more of the academic community, must be engaged alongside technical assistance and development work in order to facilitate and push understanding. Moreover, this would foster consensus on priority questions to ensure credible evidence and sound results based management. The importance of sharing results with others in the community as well as with in-country peers was also underlined from shared experiences of conducting small counterfactual studies.
As the research community continues to build on impacts assessment of certification, the lively and informed debate from this session indicates the many issues and solutions that will continue to be explored across the monitoring and evaluation landscape in future years.
To download resources from this and other sessions at the ISEAL Conference Public Day visit our conference resources page.