Measuring Loyalty Program Activation & ROI

How To Measure Your Loyalty Program’s Incremental ROI

by Mickey Neuberger
Vice President, Loyalty Strategy
TIBCO Loyalty Lab

Prior to launching a loyalty program, smart marketers build ROI models that forecast incremental profits based on anticipated lifts across three key customer revenue variables: average order size, yearly purchase frequency, and yearly retention rates. These models make assumptions on funding, breakage, and participation rates to estimate results.

One year later, post-launch, those same smart marketers are lost when asked to prove that the program is driving the incremental results they promised. That’s because measuring true incremental lift is very hard. How can you tell what loyalty members would have spent if no program existed? The approaches described below are some proven methodologies that we’ve used to help our clients approximate incremental lift and thus understand their loyalty program’s impact. And while no one technique is perfect, performing a few of these analyses concurrently provides a very accurate range.

Compare Members Versus Nonmembers
First, marketers should simply measure the revenue performance of loyalty members versus nonmembers. Clearly, we expect to see some self-selection as a higher proportion of best customers enroll and participate in the program. Thus, a gap in frequency and order value between members and nonmembers should exist. Even though marketers can’t attribute the entire gap to incremental lift, it does provide a quick, easy-to-calculate read on the upper range for approximating incremental lift. More importantly, negative to zero gaps between members and nonmembers present a “red flag.”

New customers are a particularly good segment to break out for high-level comparison. For one client, we observed a 100% increase in the number of shoppers who came back to make at least a second purchase in the new member loyalty group versus nonmember group (within 60 days of their first purchase). As discussed, it’s impossible to quantify the portion of that 100% increase due to pure lift (or to self selection), but having your best customers self-identify is incredibly useful for targeted marketing efforts. Clearly this is a group who will be most receptive and will yield the highest return on marketing investments. Furthermore, the 100% uptick provides an approximate ceiling for what the incremental lift might be.

Isolate And Compare Like Groups
To avoid the self-selection bias, the single key is the ability to isolate similar groups of customers for comparison. For companies that already have strong customer capture (e.g., online/catalog companies and other retailers/travel/hospitality companies that have strong customer identification), pull a relevant subset of nonmembers that exhibit nearly the same pre-enrollment behavior of the loyalty member group. For instance, if the average frequency of the member group prior to enrollment was 1.7 purchases and $125 average spend annually, then select a group of nonmembers with the same metrics. Next, with this rigor, compare behavior post enrollment for the loyalty member group versus the nonmember group. This will clearly illustrate the lift most directly correlated with the loyalty program.

For example, for the same client mentioned in the previous “Compare Members versus Nonmembers” section, we performed the following analysis:

• Identified every shopper, loyalty program member and nonmember that made at least one purchase over first three months of loyalty program. For the member group, the purchase had to be made within 24 hours of registering for the loyalty program.

• The two groups above were further filtered by only selecting shoppers who had made at least one purchase in the 60-day period prior to the purchase identified in step one. This effectively filtered to “recently” active shoppers.

• The two groups were further filtered by only selecting shoppers who had made at least one more purchase PR IOR to the 60-day period before the date of purchase used to identify the shopper in step one. In other words, the shopper could have made a purchase 61 days before his “qualifying” purchase (step one), or two years before, or any time between. This step was performed to ensure we had sample shoppers that were also “historically” active shoppers.

• The effect of performing steps 1-3 was to establish two statistically significant cohorts of extremely similar, established, “best” shopper cohorts (historically and recently active loyalty members and non-loyalty members).

• To further validate the comparability of these two samples, we compared the average expenditure per group during the 60 day period prior to their “qualification” dates (from step one), and the averages were statistically identical.

• Finally, we observed the nonmembers’ group 60-day post expenditure average (from date in step one) to set an expected average expenditure for the loyalty member cohort. This was then compared against the actual loyalty member’s 60-day post expenditure average which was 39.25% higher. Basically, loyalty members spent 39.25% more than what our control group predicted they would spend – the direct result of the loyalty program.

• Note, this analysis, in concert with the conclusions from this client’s “member versus nonmember” analysis described in the previous section, indicates that the program’s incremental lift is in the range of 40-100%. One caution with this approach is that by filtering to a like set of customers (active customers in the example above), you are making conclusions on the overall program’s impact based on a subset of customers that are not necessarily characteristic of the whole group.

Holdout A Control Group
The purest way to measure incremental lift is to randomly assign every existing and new customer to a control group, which will never be exposed to the loyalty program. Then invite all non-control, pilot members to participate. Post-launch, the marketer can measure incremental revenue from the pilot group versus the control group. While this method provides the most accurate read, it too has some issues. First it can only be done in the online world, since there is no reliable way to maintain an unexposed group in physical stores. Additionally, having to maintain a control group that is never exposed to the loyalty program diminishes your marketing efforts. You can’t market the program heavily on your home page, you can’t run seasonal loyalty program promotions, CSR s can’t promote the program, etc.

Another approach, especially for retailers with physical stores, is holding out a market for comparison – essentially piloting the program in order to see the overall impact on purchase metrics in select markets. You can then compare performance across markets using historical results as a baseline. This, too, has a key shortcoming: it’s very hard to separate out what the market’s impact on performance was versus the loyalty program’s (e.g., maybe the improved weather in San Francisco, not the loyalty program, is why sales spiked?).

Conclusion
There is no perfect answer to “what loyalty members would have spent if no program existed?” However, by employing several of the techniques described above, marketers can take a very educated guess at a range. This allows them to assess the overall impact of their program and informs future plans.