Why and How to Measure Social Impact and Outcomes: A Discussion with Jeffrey L. Sturchio

Jane was given $250 million to launch a national multi-media marketing campaign for her company’s newest household detergent. At the mid-point of the campaign she looked at the numbers and saw sales were doing great in the Southwest but poorly in the Northeast. She investigated further, made some changes and course-corrected. At the end of the campaign, Jane could report a positive ROI and specific impact on sales to the CEO, enabling her to obtain another round of funding. 

Sounds logical right? Then why do so few companies measure the ROI and impact of their corporate social responsibility (CSR) initiatives with the same level of rigor? That is the question on the mind of Dr. Jeffrey L. Sturchio, CEO of Rabin Martin, a global health strategy consulting firm that helps a wide range of clients in the biopharmaceutical industry with initiatives in global health and CSR. I sat down with Jeff recently to chat about the importance of measurement in CSR and who is leading the way. 

MK: Why is it so critical to measure the outcomes of corporate social responsibility (CSR) initiatives? Aren’t they “doing good” by their very nature?

JS: Whether you are working on the CSR or commercial side of business, you must think carefully before you start (a project) about what you’re trying to accomplish. No self-respecting business would think of operating without having financial metrics in place to be able to tell if their sales and marketing efforts are achieving targets or if their people are meeting their objectives. It should be the same in CSR: Are your CSR programs helping you to accomplish what you are trying to do? Are social business initiatives delivering on intent?

It should be accepted practice in CSR to design a monitoring and evaluation (M&E) framework for every project. This shows that you have the same discipline and rigor that the other parts of the business do. It demonstrates you are serious about the work. And it allows you to speak the same language as the other decision makers in the company in terms of ROI and impact. If you can monetize the work you are doing, you can go to leadership with a persuasive case for additional resources to invest. 

Another reason effective measurement is so important is that it’s the only way you will know if a program is well-designed and to allow for mid-course corrections when needed. What you expect to happen when you design a project will rarely be what actually happens when implemented. But if you have an M&E framework, you can see if you are on track or not and the information allows you to adjust. One company that is doing this well is Novartis (see sidebar below). 

Another is Merck with Merck for Mothers (MfM), an initiative launched in 2011 to help eliminate preventable maternal mortality worldwide.  (Rabin Martin has worked closely with MfM since the outset.)  Last year, Merck for Mothers published its first Research Compendium with links to more than 100 publications it has supported over the past six years. The questions these publications explore and the answers they uncover have been invaluable in informing the initiative’s work. It only resulted because there was a strong M&E framework at the project level from the beginning. 

Measuring Impact: Novartis Access Program: One company that has put a rigorous M&E framework in place to measure the outcomes of its CSR efforts is the Swiss pharmaceutical giant Novartis. It is working with Boston University to evaluate the impact of Novartis Access, which offers a portfolio of non-communicable disease (NCDs) medicines at a price of US$1 per treatment per month to public and NGO customers in low-income and middle-income countries. Starting in 2016, study investigators conducted a cluster-randomized controlled trial in eight counties in Kenya. As presented in the Lancet Global Health earlier this year, results from the initial baseline showed that “after 15 months, [Novartis Access] had a positive effect on availability of amlodipine and metformin at facilities, but had no effect on availability of the other medicines in the Novartis Access portfolio.”  It noted, however, that “access programs operate within complex health systems and that reducing the wholesale price of medicines might not always or immediately translate to improved patient access. The evidence generated by this study will inform Novartis's efforts to improve their program going forward. The study also contributes to the public evidence base on strategies for improving access to medicines globally.”

MK: How do you see most companies measuring their CSR programs today?

JS: Too many companies spend millions of dollars on programs and hardly anything on measuring impact. That can result in throwing good money after bad if you expand a program that isn’t actually working. Instead, people talk vaguely about how important a project is and that it will ultimately be good for the company’s reputation. But they can’t be exact about the extent to which the program is helping its intended beneficiaries or talk about the timeline for a reasonable return on investment. This lack of precision about ROI puts CSR executives at a disadvantage to their business colleagues when senior management has to make decisions about resource allocation.

MK: But companies are reporting to some degree on their CSR programs, aren’t they?

JS: Yes, but most CSR metrics focus on inputs – e.g., how much money did we invest? Some companies go further to look at outputs – e.g., how many products did we distribute, how many health care workers did we train? This shows what you got for your investment. But few companies think about the outcomes and impact their investments actually had – if, for example, they have improved the health and wellbeing of the people they were supposed to reach. 

A good example of this is a pharmaceutical company we worked with that was trying to address hypertension in sub-Saharan Africa. They spent a lot of time looking at how many people they screened for high blood pressure, how many people were diagnosed, and how many were put on treatment. But they didn’t place a similar effort into capturing the ultimate impact of their program – were they and their implementing partners reducing the number of strokes from hypertension? Were the people using their anti-hypertension medicines (or others) able to go back to work and be more productive? 

MK: Many CSR initiatives that tackle tough social issues are conducted in complex environments with many intertwined factors. In such environments, how can measurement show causality of a program’s intervention? How can  you  build an evaluation process that allows you to claim appropriate attribution for your program or intervention?

JS: Skeptics will  often say that your intervention won’t solve the problem because there are all these other contributing issues.  But, if you are thoughtful – by examining and understanding the health care ecosystem in the country you are in and what’s connected to what – it is possible to understand and measure the contribution of your intervention. It’s simply nihilistic to say at the outset that you can’t possibly  measure the impact. In fact, you have an obligation to try to measure your contribution for at least three reasons:

1. To illustrate to other stakeholders that you’re making a positive contribution, 
2. To persuade your internal colleagues that this is a good way to use resources, and 
3. To know what you’re doing right and what you’re doing wrong. 

If you  have a robust M&E  framework – even if you can’t be 100 percent specific about what impact your program is having  – it still adds to the knowledge base. You will undoubtably find other problems going on in the health care environment that need to be fixed and this will help reach the overall goal you and others are trying to attain. That is the unintended consequence of M&E programs – they uncover other information on how health systems operate and over the long run this leads to improvements in population health overall. 

A good example of the work being done in this area is  by Boston University and leading global pharmaceutical companies through the Access Accelerated project. (See side bar below).

MK: How should an organization get started measuring impact? 

JS: You must design the M&E framework before you start the project. You can’t start it mid-course. Companies should start by thinking about what it is they are trying to affect and work back from there using a logic model that specifies inputs, outputs, outcomes and impacts.  You need metrics along the project’s entire trajectory. And, you must have a system to collect the data in a reliable and sustainable way, so you’ll be able to analyze it. 

Access Accelerated: Through the Access Accelerated initiative, 26 biopharmaceutical companies are tackling the growing burden of non-communicable diseases in low and middle-income countries by co-creating scalable and sustainable solutions to improve people’s health. A team from Boston University is working with the companies to integrate a robust measurement framework into the project to serve as a common language for categorizing, understanding and comparing access programs. The framework includes three main components: 1) A taxonomy of 11 strategies that describes common approaches used by access programs; 2) A series of logic models—one for each strategy—detailing the pathways by which programs may achieve impact; and 3) A set of clearly defined indicators for reporting program activities and achievements. The team has developed the Access Observatory, an online public repository of information on access programs, structured according to the measurement framework. To date, the Observatory has compiled reports on 63 programs in 103 countries.  For more information, see their 2018 report.  

MK: What types of skills does Rabin Martin have to help clients develop  M&E frameworks?

JS: We bring a unique mix of skills to bridge research, evaluation, programming and policy, with an unwavering commitment to measurable results.  We have experts in public health program design and implementation, often with hard-won experience on the front lines establishing and operating programs. They understand what you need to do to collect data and analyze it. We also have people who understand the art of program evaluation and the nuts and bolts of managing an M&E framework. It helps to have done this kind of work when counseling clients on the best approaches for building M&E into their program designs at the outset – and to do so in a nuanced way.

MK: How do you ensure that impact measurement becomes a means to an end rather than an end in itself?

JS: M&E must be part of the discipline of how you manage your entire portfolio, with an emphasis on becoming a learning organization. It is a mindset of looking quantitively at data to inform decision-making. Everything you do is a question of allocating scarce resources to accomplish what you are responsible for – in that context, without the right data, it’s not possible to tell which investments are yielding the most health for the money. There is truth in the old cliché that you can’t manage what you don’t measure.