Evaluating impact through a Most Significant Change approach
From an article by Better Evaluation
Sometimes it is difficult to evaluate the impact (and unintended impact) that your organisation is making as this story shows:
In 1994 Rick Davies was faced with the job of assessing the impact of an aid project on 16,500 people in the Rajshahi zone of western of Bangladesh. The idea of getting everyone to agree on a set of indicators was quickly dismissed as there was just too much diversity and conflicting views. Instead Rick devised an evaluation method which relied on people retelling their stories of significant change they had witnessed as a result of the project. Furthermore, the storytellers explained why they thought their story was significant.
If Rick had left it there the project would have had a nice collection of stories but the key stakeholders’ appreciation for the impact the project would have been minimal. Rick needed to engage the stakeholders, primarily the region’s decision-makers and the ultimate project funders, in a process that would help them see (and maybe even feel) the change. His solution was to get groups of people at different levels of the project’s hierarchy to select the stories which they thought was most significant and explain why they made that selection.
Each of the 4 project offices collected a number of stories and were asked to submit one story in each of the four areas of interest to the head office in Dhaka. The Dhaka head office staff then selected one story from the 16 submitted. The selected stories and reasons for selection were communicated back to the level below and the original storytellers.
Over time the stakeholders began to understand the impact they were having and the project’s beneficiaries began to understand what the stakeholders believed was important. People were learning from each other. The approach, called Most Significant Change, systematically developed an intuitive understanding of the project’s impact that could be communicated in conjunction with the hard facts.
Rick’s method was highly successful: participation in the project increased; the assumptions and world views surfaced, helping in one case resolve an intra-family conflict over contraceptive use; the stories were extensively used in publications, educational material and videos; and, the positive changes where identified and reinforced.
The Most Significant Change (MSC) approach involves generating and analysing personal accounts of change and deciding which of these accounts is the most significant – and why.
The are three basic steps in using MSC:
-
Deciding the types of stories that should be collected (stories about what - for example, about practice change or health outcomes or empowerment)
-
Collecting the stories and determining which stories are the most significant
-
Sharing the stories and discussion of values with stakeholders and contributors so that learning happens about what is valued.
MSC is not just about collecting and reporting stories but about having processes to learn from these stories – in particular, to learn about the similarities and differences in what different groups and individuals value. It provides some information about impact and unintended impact but is primarily about clarifying the values held by different stakeholders.
By itself, MSC is not sufficient for impact evaluation as it does not provide information about the usual experience but about the extremes. It works best in combination with other methods for gathering, analysing and reporting data. It doesn't provide comprehensive information about the impacts produced by an intervention.
MSC is particularly useful when you need different stakeholders to understand the different values that other stakeholders have in terms of "what success looks like" - criteria and standards for outcomes, processes and the distribution of costs and benefits. MSC can be very helpful in explaining HOW change comes about (processes and causal mechanisms) and WHEN (in what situations and contexts). It can therefore be useful to support the development of programme theory (theory of change, logic models).
Read more here - includes several resources.
Retweet about this article:
From an article by Better Evaluation, 13/09/2023