Experiment to Discover Your Positive Social Impact
Part II of a series on social impact for data scientists
In part I of this series, I outlined a process of how you can find your own ways to have a positive social impact as a data scientist. One core idea is to experiment with different approaches to find out what works for your needs and situation.
In this blog post, I dig deeper into how we can run experiments that yield insights into these questions.
The “Try-Measure-Learn” loop
The “try-measure-learn” loop is a simple process for conducting experiments. It was proposed in Eric Ries' book "The Lean Startup" and adapted for social impact work by Ann Mei Chang in her book "Lean Impact".
This loop will not meet the standards of scientific rigor, but it is more suited to casual learning from experience.
First, you define your assumptions with the resulting expected outcome and try the task you want to experiment with. Then, measure the actual outcome of our task. Finally, draw conclusions based on what you did and measured to learn whether your task lead to what you wanted. This in turn informs your next experiment.
Let’s look at an example. Remember Alina? Her partner Roberta is working at a crisis help line and answering around 10 calls per hour. She and her colleagues each have specific areas of expertise which enables them to handle more serious concerns. To figure out which helpline operator will handle which call requires a complicated screening process.
Roberta feels that a different screening process could help them reduce the time spent screening and allow them to help more people overall. Alina recommends to Roberta to run an experiment before changing the process for all operators to validate the outcome of such a change.
Roberta and two of her colleagues decide to try the new process for two weeks. They are measuring for those two weeks how long it takes them to identify which colleague should handle a more serious call. Unfortunately, the average time it takes them is longer than usual: 10 minutes instead of 7. So they are learning from the experiment that changing their process is not an effective way to shorten their screening times and answer more calls on the crisis help line.
According to Chang, for achieving social impact, we need to validate three kinds of assumptions with this loop: Acceptance, growth, and impact assumptions.
1. Validating acceptance assumptions
First, validate the acceptance of your contribution by the organization and by yourself. If nobody wants to use what you provide, there can be no positive impact whatsoever. Likewise, if you hate doing it, it is not worth continuing, either.
Chang recommends starting with contributions of a fixed scope to gather basic data on acceptance. At the risk of sounding a bit cheesy, we can call that a "Minimum viable contribution". (The emphasis here is on viable. If you don't achieve acceptance, it is not a viable contribution!)
Guiding indicators for this stage can be
Did you manage to reach any beneficiaries with your approach?
Do your beneficiaries use what you provide?
Do they like using it?
Do they recommend it to others?
Do you enjoy doing this?
Does this approach fit your values?
Do you personally feel doing this approach is meaningful?
Do your supporters enjoy working with you?
2. Validating growth assumptions
Second, validate the growth and sustainability potential of your contribution. Find out how long it could take to reach the impact you want. Find out who else is facing the same issues as your first beneficiaries. What would it take to reach them?
Based on your first contribution, do some planning exercises. For example, this could be estimated cost and effort projections. Find out where you don't have enough insights yet to know whether you can sustain and grow your efforts.
Guiding indicators for this stage can be
Do you have access to the resources necessary to keep your approach running: Time, funding, collaborators, etc.?
Can you scale it up to other beneficiaries, locations, contexts?
Can your contribution help to eventually reach everyone affected by the issue?
3. Validating impact assumptions
Finally, validate the impact you can achieve with your contribution. You will want to refer to the impact goals you defined as part of your strategy here.
There are many ways to evaluate impact, from very rigorous to more pragmatic. The most important distinction here is between inputs (the resources you use), outputs (what you do or deliver), outcomes (the effects on your beneficiaries) and impact (the effects on society). These four can be modeled as a chain called IOOI-chain.
As an example, let’s take a vocational training program for unemployed young adults.
Inputs could be: Teaching staff, classrooms, computers, books, lunch food
Outputs would be: The number of trainings delivered
Outcomes could be: The number of students who graduate from the program, the percentage of graduates that find a job within three months after graduation
The impact could be: Youth unemployment in the area drops by 3%
It makes sense to measure all of these four elements of your IOOI-chain to understand the relationships between them better.
Again, here are some guiding questions to get you started, each with a simple way to evaluate them.
Outcomes: Did your beneficiaries change their thinking or their actions? You can measure those with surveys — before exposure to your contribution and afterwards.
Outcomes: Did the situation of your beneficiaries improve thanks to your contribution? Again, you can measure those with surveys before and after exposure to your contribution
Impact: Is there a noticeable shift in society? This is the hardest to measure and it takes the most time for impact to manifest in society. You can use publicly available statistics on the extent and development of the issue you are trying to affect.
Generally, for short-term experiments, you can focus on measuring outcomes instead of impact. You need to have a theory of how your outcomes lead to broader impact, but you might not be able to measure it for small experiments.
Ready to start your Experiments for social impact?
I hope this blog post encouraged you to conduct your own experiments! Please do let me know via Twitter or email if you did and what you learned. In the next part of this series, I will share my learnings from conducting my own experiments within the last 15 years.