Measurement & Optimization
Impact measurement asks what a decision changed compared with what would otherwise have happened. It can inform future choices, whether the outcome is customer retention, profit, or a broader social or environmental effect.
THE IDEA
Measurement & Optimization
Assessing a decision requires defining the outcome and asking what changed because of the action. Impact can concern broad social, environmental, or economic effects, or a narrower business outcome such as customer retention.
Broad effects can be difficult for an individual organization to measure directly. Definitions may be contested, many actors influence the result, and useful evidence may require a long study. For example, an organization reducing fossil-fuel use can track changes in its emissions more readily than isolate its contribution to global temperature change. Tracking an outcome remains useful, but it is different from demonstrating a causal impact on the wider system.
For a customer-acquisition campaign, the incremental effect is the number of customers acquired because of the campaign. Counting every new customer during the campaign overstates that effect if some would have arrived anyway. The business assessment also needs to compare costs with incremental benefits, using an appropriate measure such as profit rather than treating additional revenue as profit.
Reliable impact estimates can help compare future actions. When such estimates are unavailable, predictive models can inform the choice,15 but their forecasts still depend on assumptions and the data used to build them. A prediction of customer behavior does not by itself show how that behavior would change under an intervention.
An organization may need additional data to assess alternatives. The useful question is which missing information would improve the decision. A ‘360° view of customer’ aims to combine relevant customer information; collecting a wider range of data is valuable only insofar as it improves the analysis.
When to use it
Deciding whether to invest in a new customer relationship management system by estimating incremental customer retention.
Evaluating the effectiveness of a social media campaign on brand awareness using controlled experiments.
Choosing between two energy sources for manufacturing by assessing long-term environmental and economic impact.
Optimizing a recommendation algorithm by measuring incremental revenue from personalized suggestions.
FROM THEORY TO EVERYDAY LIFE
In practice
Marketing Campaign Impact
A company randomly assigns eligible customers to an advertising group and a comparison group. Comparing acquisition outcomes helps estimate how many additional customers the campaign produced, rather than crediting it for every acquisition during the period.
Sustainability Initiative Impact
A company invests in reducing coal dependency. While immediate benefits include reduced emissions, the ultimate impact on global warming is difficult to measure directly, so tracking key outcomes like emission levels may suffice.
A MOMENT TO REFLECT
Put it to work.
How can you distinguish between incremental impact and overall results in your own business decisions?
When is it appropriate to measure impact at a macro level versus focusing on key outcomes?
What data would you need to build a predictive choice model for a common decision in your organization?
CONTINUE THE CONVERSATION
Source texts
- Evidence-Based Decision-Making How to Leverage Available Data and Avoid Cognitive Biases (Andrew D. Banasiewicz)