Validity of the Net Promoter Score Model in Business
As originally published on May 12, 2012
The net promoter score (NPS) has been widely touted as a leading indicator of customer loyalty resulting in increased customer sale (be it repeat, new, or regained). In Part I of this blog series, you will see how one should be validating the NPS mathematical model and ensure that the efforts to seek this score, let alone improve it, are really impacting the business needle in a positive manner. The scope of this blog is around published literature as well as some forward thinking on the relevant NPS process.
In a recent publication, on NPS we note the following conclusions on NPS and its benefits as a leading indicator. The mathematical model was in the form as shown below:
% Business Growth =f(NPS) where NPS is defined as:
NPS= % Promoters – % Detractors
This data is compiled in a survey with customers who have recently engaged in a sale experience with the business resulting in some purchase.
- Promoters (9-10) are those customers that rate the business or sale experience highly (likely to engage in a repeat business and/or refer to someone else who could be influenced to engage in a new or repeat or regained purchase with the same business).
- Detractors (0-6) are those customers that rate the business or sale experience very poor and are not likely to engage in a repeat sale or refer someone and possibly even influence someone to avoid a repeat or new or revisit sale experience with the business.
- Passives (7-8) are remaining customers who are neutral in rating their latest sales experience.
Thus NPS scores can also bear negative values when % Detractors is > % Promoters.
The assumptions for this model are as follows:
- The regression is linear – there is a direct relationship between the NPS and increased growth in business.
- Measurement System Error:
- Reproducibility: The measurement system around the survey itself is valid meaning that the NPS would be consistent even with a repeated survey or measurement.
- Repeatability: There is no significant error in the NPS if repeated across different customers.
- Correlation and its Causation:
- Increased business growth is directly influenced by an increase in NPS and not by chance.
- Sample Size:
- The minimum number of NPS survey data points needed in this one variable mathematical model is 10 data points without even considering replicates.
- Confounding Variables:
- Assuming that there are no external factors influencing the business growth in this study/analysis.
- Other Changes:
- Technology is not influencing an increase in new customer purchases.
- Residual Diagnostics:
- The four block residual diagnostics diagram is executed using the standardized residuals.
Discussions and Conclusions:
We will now look at each of these components of a liner regression model to validate its relevance.
As we all know and accept that something like NPS is definitely advantageous for businesses to grow by leveraging with a leading indicator or predictor. NPS also allows us to have a highly reliable forecasting system. Once we have the NPS we can then predict customer behavior and thus the business growth.
When we have predictions on growth, our forecasting for sales improves and we bear the least residuals between the predicted and actual sales figures. The lowest residuals on predicted and actual sales allow us to achieve two aspects in our business:
- The least probability for missed deliveries – OTD (if our forecast is lower than actual sales)
- The least idling inventory – ICC (if our forecast is higher than actual sales)
|As users of this model from Satmetrix, it is our responsibility to ensure that the models we apply to our business process reflect the process in reality. This allows us to make the proper investments without any significant negative impact to the business. If we forecasted a high increase in growth due to a high NPS we should plan and schedule a high volume production accordingly. On the other hand if we forecasted a low volume due to a lower NPS we should then be prepared to relieve our inventory and accordingly tailor the supply chain functions to control our raw material inventory turns.|
- I think that I would like to use this system frequently.
- I found the system unnecessarily complex.
- I thought the system was easy to use.
- I think that I would need the support of a technical person to be able to use this system.
- I found the various functions in this system were well integrated.
- I thought there was too much inconsistency in this system.
- I would imagine that most people would learn to use this system very quickly.
- I found the system very cumbersome to use.
- I felt very confident using the system.
- I needed to learn a lot of things before I could get going with this system.
The SUS is based on the queries posted above and the ratings by the customer who interprets the query and rates it accordingly on a Likert Scale. The model based on SUS and NPS claims aR2of 0.36 with an “r” value (correlation coefficient) of 0.61. R2is the regression coefficient in the NPS and SUS model.
The linearity is of course tested by the R2value however; the following measures also need to be investigated to validate the model:
Residual Diagnostics Test Plots of:
- Histogram of residuals shouldn’t show any unusual visual trends away from normality
- Normality of the Residuals – Needs to follow a normal distribution (p value > 0.05).
- Standardized Residuals vs. Fitted value of NPS should show all points within ±2.
- Standardized Residuals vs. Order of Observations should be random and in control.
This is not presented in any of the published literature on topics relating to the NPS or NPS-SUS model. A sample of the residual diagnostics plot is shown below:
Measurement System Error:
The measurement system error associated with the input values of NPS and SUS in the model is suspect as of now. Since the inputs of NPS and SUS are derived from a survey type effort there is a high chance of a measurement system error. Again, this error identification and quantification helps us understand the validity of the inputs.
Correlation and its Causation:
The concept that correlation doesn’t mean causation is addressed by the residual diagnostics plot. Additionally, the model when tested will yield a predicted and actual growth value which will allow us to validate the causation and correlation. There has been no mention of a prediction system as all growth values have been plotted in a reactive than predictive mode. Even past the residual diagnostics it would be vital to pass the confounding variable aspect discussed below.
The sample size in one of the SUS claims using 146 responses, however, most NPS-Growth data plots have used anywhere between 6 to 15 data points. The 146 sample here refers to the NPS-SUS model between loyalty and NPS not the actual growth and NPS.
The confounding variable is one that is acting behind the scene and not taken into effect. For example there are several factors influencing the growth in business (measured by increased revenue):
- Product Mix
- Product Pricing and associated Discounts
- Technology Intervention and Changes in product design and performance
- Product Integration sold as a package
- Similar Product Comparison before and after NPS improvement
- Economy of the region registering sales such as recession or boom
- Affordability of customer making the purchase
- Demographics of purchasing “customer”
Without looking at these factors or lingering variables, it would be difficult to establish the causation and correlation concept as well as validating the predicted versus actual growth as a result of direct increase in NPS.
- 2012 Predicting Net Promoter Scores From System Usability Scale Scores
- 2011 Introducing: the Net Promoter system
- 2009 Net Promoter Score Defined by Derrick Daye Managing Partner, Brand Consultant
- 2008 How the Net Promoter Score (NPS) Can Drive Growth: The Economic Advantage of Superior Customer Relationships
- What is a Net Promoter® system and how does it work?
- What is a Net Promoter® system and how does it work?