Our Methodology

Why and How We Measure Impact at New Story At New Story we measure impact to quantify how a home creates change, improve our work and learn what to iterate on to optimize our impact. Do we have to do this? No. Lots of organizations don’t. Many nonprofits will only tell you the number of scholarships given, people helped, vaccines delivered, etc. We know the true impact is much deeper than these “vanity” metrics. We’re hyper focused on improving our model and behavior change, so doing our best work involves measuring our impact and iterating to improve. Our Methodology (how we measure) A home impacts many aspects of quality of life. At New Story, we collect impact data via a household survey, given once every 6 months to a head of household in the family. Through our survey we track impact metrics in the following indicators: Health & Wellbeing Economic Opportunity Education Community +Quality of Life +Occupation +Highest Level of +Changes made to the +Satisfaction +Previous Home +Education Attained by home +Stress Ownership Heads of Household +Time Spent at Home +Primary Source of +Household Monthly +School Attendance (El +Accomplishments Drinking Water +Income (El Salvador Salvador Only) +Benefits +Water Purification Only) +School Performance (El +Community Changes +Health Concerns +Household Assets Salvador Only) +Safety +Frequency of Illness +Privacy +Sleep Quality +Security +Ideas for Community Improvements Lean Data We use a Lean Data approach pioneered by our friends at Acumen to collect actionable data from our beneficiaries. Here’s why we choose to collect impact data via a Lean Data approach. ● It’s human & beneficiary centric - our survey takes less than 30 minutes to administer, is designed to be conversational and gives beneficiaries the opportunity to share their experience with our donors. ● It’s decision centric - we use a mixed-methods approach to collect quantitative and qualitative data to not only track impact metrics but learn how we can improve our process and operations. ● It’s fast - we collect data via a tablet or mobile phone that syncs with wifi. So we can get data as close to real time data as possible. ● It’s lean - While the name “lean data” might have implied this… , using low-cost technology means we can easily iterate on survey questions with our local partners and update them for instant use.

Now, you might be asking...what about randomized control groups? Where is your observational before-and-after study with a convergent-parallel mixed method design? (Isn’t academic jargon fun?) Here are some of the reasons we DON’T use traditional “monitoring and evaluation approaches” right now. 1 Randomized Control Trials - RCT’s are an experimental approach to measuring the social change in a policy or intervention. (Think randomly giving houses to one group of beneficiaries and not to another and then surveying both groups to assess the change in quality of life over time.) 2 Why we don’t do an RCT (right now) . They’re expensive - think hundreds of thousands of dollars and waiting years to see data and results. As a growing organization our approach is centered on the end-user. Right now, we’re focused on building the best houses possible through a community process. It’s not a feasible priority to go back and survey families who haven’t received housing yet still live in slum communities. In this case, our end-users are some of the most vulnerable and impoverished women, men and children in this hemisphere. When optimizing data to understand value creation and areas for operational improvement, we don’t think it’s a smart use of time and money to go back and survey non-selected families. However, we do believe that there are ways to optimize diligence in our methods to make sure our data collection process and data is as scientifically rigorous as possible. Output Only (Laundry List) Approach This is a laundry list of program outputs such as number of homes funded, number of families living in homes, number of women and children helped. This list is often required by funders, but doesn’t go any deeper to help us understand the behavioral and environmental changes created by giving someone a home? Did it help someone create a business or feel trust their neighbors? Did it actually lift a family out of poverty or create a thriving community? Working with a local partnership model, this approach can also create a top-down compliance culture rather than a bottom up optimization of impact from all stakeholders. (Note: we do track some of these numbers, you can see many of them here. We just don’t think it should be the only or most important metric to track impact.) 1 New Story does not believe in “monitoring” and “evaluating” our beneficiaries. If our goal is to create thriving communities by learning, iterating and improving our impact along the way, there are many methods to optimize for value and impact. 2 Randomized control trials are the gold standard because they are most accurate scientific method to evaluate causal effect. They are not the only indicator of optimizing value and impact. They are often more feasible for established organizations operating at scale. They do not however take into account the dynamic nature of humans conditions in slums and growing organizations where products and processes are changing to optimize efficiency and value. We are excited, however, about the possibility for a quasi-experimental study in the future through a pipeline approach. This approach would allow us to optimize construction timelines to utilize families who have yet received housing as a control group. More on this soon.

How We Maintain Rigor In Our Process. We’re invested in making sure we have the best quality data for decision making which is why we: ● Minimize selection bias by hiring external Research Assistants to administer household surveys. ● Work with our advisors and local partners to track metrics from previous housing studies, build on current research and localize surveys. ● Use mixed methods design so we are measuring both quantitative and qualitative metrics within each indicator to track our impact. Why We Measure: There’s a difference between measurement of causal effect and optimizing for value creation. While randomized control trials are the gold standard for measuring causal effect, as a young organization we are hyper focused on value creation for our end-users, beneficiaries and donors. Because creating social change requires behavior change we are deeply invested in understanding first what behaviors are changing to set us up to measure how much they are changing. Our team spends a tremendous amount of time (all of our time, actually) refining our process to create thriving communities for those most in need. Tesla wasn’t the first company to build cars, they just used a new model and process to do it differently. Similarly, we aren’t the first organization to build houses for vulnerable communities. From local partnerships to participatory design, how we build homes and communities matters. Impact data is just one part of our process equation that helps us deeply understand our end user, refine our model and optimize for impact. We’re constantly improving our model to maximize for impact. Stay tuned for our initial case studies later this year. Want to know more? I’d love to chat, you can reach me at [email protected]