Report on lessons learnt about research design and methods

This methodological paper from ISEAL shares insights and lessons learned from three ongoing impact evaluations that completed their baseline in 2016 and are due for end line evaluation in 2019.

These baseline studies are part of the Demonstrating and Improving Poverty Impacts project and were undertaken in the cotton sector in India and the coffee sector in Kenya and Indonesia.

This paper attempts to address fundamental conceptual questions that arise in the course of designing and undertaking impact evaluations of sustainability standards and to share cross-cutting learning from how the DIPI evaluations addressed them. These include questions such as:

  • What is the role of theory-based evaluation in this field?
  • How do we understand and study ‘treatment’ in the context of standards?
  • What are selection effects and why should they matter?
  • What constitutes credible counterfactuals?
  • How and when can randomisation be achieved?
  • What is meant by mixed methods?

We focus on these questions as we consider them to be fundamental concepts that every impact evaluation in this field will encounter, irrespective of standard, sector or geography. A focus on these questions will also help strengthen the approach and robustness of impact evaluations undertaken by standards.

The observations in this methods learning paper are targeted primarily at standards systems, and specifically their monitoring and evaluation (M&E) teams, but it is hoped that researchers and others will find it a useful read. Ultimately, our hope is that this is not read as a paper purely about methods and approaches to evaluation but helps understand the systems that we attempt to evaluate at a more fundamental level.