The untold secrets of B2B SaaS analytics: When to say no (Part 1)

Since there is so much buzz around data these days in B2B SaaS companies¹, especially with the latest insurgence of AI, many business professionals often turn to their analytics colleagues in search of insights that will help make the right move and do the right thing.

Sometimes, that’s exciting because the business really does need data to make the right choices. But sometimes, data can be just a scapegoat for many other unaddressed issues in the business: lack of alignment, lack of communication, lack of clear strategy, overcomplicating, and sometimes even laziness.

What no one ever told me was that a big part of being a strong analytics professional in these situations is not only knowing how to build the right data models and do the right analyses, but knowing when not to do any analytics whatsoever, and when to say no to data projects.

To be more precise, if you can evaluate early on whether a potential analytics project will truly make an outsized impact on the business, saying no to low-value projects is actually much better for the company in the long run. It makes you a strategic thinker and a more valuable asset to the business.

That’s why I want to share my approach of evaluating potential analytics projects. It’s not perfect by any means but it has served me reasonably well in my day-to-day job.

It consists of four steps:

• Step 1: Check whether you even need analytics to make a decision
• Step 2: Determine if the business stakeholder(s) will actually use your work to make a decision
• Step 3: Evaluate if you can get to (approximately) the right answer by doing a back-of-the-envelope calculation instead of full-scale analytics
• Step 4: If the project passes the first three checks, then use the benefit-cost ratio to make a final call

The first three steps are judgment calls. The fourth step is the only one that involves math and that can be a bit time-consuming at first. Once you do it a few times, it becomes much quicker, especially if you make an automated version of it in Google Sheets or Excel. I describe the first three steps in this post, Part 1, and focus exclusively on the fourth step in Part 2.

Note: This approach is based on my personal work experience at high-growth, fast-moving B2B SaaS businesses, where my focus has usually been on driving strategic business decisions with data. My perception, also, is that analytics in these environments has a higher tolerance for estimates and assumptions compared to analytics in established, process-oriented companies. Keep that in mind as you review these steps because they might not be applicable to all analytics teams.

Step 1: Check whether you even need analytics to make a decision

As counterintuitive as this sounds, the first step when taking on a potential analytics project is to figure out if analytics is even needed to make a business decision. Data is useful when the right decision is not obvious, but it can be frustratingly wasteful when it’s used for every business decision.

This happens often because people think they need to see data for everything. In itself, that frame of thinking is a good thing, because it means people are knowingly trying to be more data-driven in their decision-making at work.

The thing is, you can sometimes make a very educated decision without any data, and doing so when appropriate is one of the most powerful skills in the arsenal of any data scientist or analyst. If you can help your business stakeholder recognize that the right decision is obvious and that no analytics is actually necessary, that’s far more valuable to the business than doing an impressive, highly rigorous analysis.

Here’s an example. Let’s say you are working as a data scientist for an innovative B2B SaaS company that sells all-in-one HR software (payroll, performance reviews, taxes, etc.) to other businesses. Every year, these businesses—your company’s customers—decide whether to renew their contracts and continue with your company’s services. The company has been operating for a while, so you have at least ten years of renewal and churn data at your disposal.

The software has a specific section dedicated to taxes, and the version that’s shown to US clients is phenomenal: slick, concise, easy to use, and even entertaining. US clients love the tax section and ever since it has been introduced four years ago, they have been highlighting it as one of the features that makes your company stand out among other HR tech businesses.

But the European clients don’t have that. Their tax section was done haphazardly many years ago when your company had only three clients in the EU, and it now looks like a sad relic of the past. It’s slow, confusing, and jarringly unpleasant compared to the other sections, which are great and equivalent to the ones shown to US clients.

You weren’t aware of this difference, but it has been brought to your attention after a very data-driven product manager joined the company. This product manager is evaluating which parts of the product need to be improved, and they come to you with this question:

“I am looking into evaluating which parts of the product need to be improved to elevate the customer experience. The European version of the tax section is super janky, but I would love to understand whether it has historically impacted financial outcomes, and if we can infer historical impact on customer churn. And, if so, what was the magnitude of that churn? I know that we collect customer satisfaction data through the NPS survey, so I wonder if we can use that to draw correlation or predictiveness? Having data-driven evidence would be super helpful to decide if my team should revamp the European version of the tax section.“

All sorts of questions come to mind when you see a request like this from your business stakeholder. What if we don’t have enough NPS responses? Do we even have enough European clients to make a legitimate analysis? How do I tell them that we can’t attribute churn to just the poor quality of the tax section? In a situation like this, the wisest approach is to pause and ask yourself if the right decision is obvious, even without any data.

I would argue the decision is obvious in this case. This is a typical scenario of overthinking and overcomplicating. The European version of the tax section has to be revamped. Whether its poor user interface was ever correlated to churn is irrelevant. It is a poor business practice if clients from one market have the good version of the product and clients from another market have the bad version of the product. It also introduces an implicit customer bias based on something the client can’t control. Not to mention that it can impact the company’s reputation if this discrepancy goes on forever.

The product manager does have to decide eventually how long this revamp should take and how to prioritize it compared to other projects, but they don’t need data to know that the revamp has to happen.

Step 1 takeaway: The answer to important business questions is sometimes simple and obvious. Knowing how to deconstruct your stakeholders’ requests and how to identify decisions that need no data is an underrated analytics skill.

Step 2: Determine if the business stakeholder(s) will actually use your work to make a decision

The more common scenario is that in which the decision is not obvious. It still doesn’t mean you have to get into the data immediately. You should also suss out if your business stakeholder is even going to use the insights you generate. By “use,” I don’t mean whether they will reference your work in a conversation with an executive, but whether your insights will actually drive decision-making in the business. How do you recognize this difference? By watching out for the greatest red flag in the world of analytics—the “it would be interesting to know” statement.

Here’s an example. We’ll use again the case of the B2B SaaS company that sells all-in-one HR software. Let’s say an account manager comes to you and says that the customers (in this case, the buyers of the HR software on the side of the client company) often ask whether usage personas can be identified among their employees by looking specifically at how their employees use the optional peer feedback tool provided by your company’s HR software.

This peer feedback tool is purchased as part of a (slightly) higher-priced bundle of your company’s product, and has generally been praised by customers for its understandable, AI-powered interpretation of qualitative text. That said, it is optional, so customers do sometimes wonder if they’re getting their bang for the buck.

This could certainly be a cool analysis, you think to yourself. Maybe those employees who use the optional peer feedback tool are also more likely to have better performance reviews from their managers? Maybe they are the employees who end up getting promoted more often? This would require thoughtful analysis setup and data exploration, but you definitely see promise in the account manager’s idea. In this situation, it’s very important to probe further, and to understand how this analysis would be used in practice.

“That’s definitely a great question,” you say. “There could be a bunch of interesting correlations and potential implications there. Is the idea that the people on the customer side, who run the HR departments, would then nudge those employees who don’t often use the peer review tool to use it more frequently? In case there is a positive correlation? What would happen if there is no correlation or if it’s even maybe negative?”

“Um,” the account manager sighs, “well, I don’t know that they would necessarily do anything. That really depends on their company’s policy and how the peer review tool is implemented and on the professional relationships at the company. But, it would be so interesting to know. I have so many customers asking this question. This insight could really elevate our narrative.”

Bam. That could have been several days of work for you: work that would have resulted in no identifiable progress for your company. When something like this happens, it’s okay to say no until the business stakeholders can identify clear motivation for this type of work and a robust course of action that would follow from the analysis.

Step 2 takeaway: Being curious is fantastic, especially if curiosity is a prominent quality in your company’s culture, but you need to remember that you are getting paid to help run a business. If you establish that your work will be used to simply generate an interesting insight for someone and will never be used for any decision-making, you will waste the company’s time and money if you say yes simply because the work sounds impactful.

Step 3: Evaluate if you can get to (approximately) the right answer by doing a back-of-the-envelope calculation instead of full-scale analytics

Okay, so let’s say the decision is not obvious and this potential project will definitely drive decision-making across the business. Well, that’s it. Sounds like you should go build that predictive model, kick off that in-depth analysis, or launch that full-blown data investigation, right?

Not necessarily. A strong analytics professional will always evaluate whether most of the needed insight can be accurately approximated with a back-of-the-envelope calculation before they commit to a full-scale analytics project. This particular step has been the hardest one for me to learn, because it goes against my perfectionistic tendencies and my selfish desire to always work on something new, challenging, and rigorous. But building this muscle—the intuition to recognize diminishing returns in potential projects—has been paramount in my analytics career.

Here’s an example. We’re still working at the B2B SaaS company that sells all-in-one HR software. The optional peer feedback tool is still the hot topic among customers, and this time, another account manager comes to you to ask for some data. They are elated because one of their accounts is so close to signing a multi-year, 5-year renewal deal. Unprecedented! But—the buyer on the customer account would love to know how this peer feedback tool thingy works longterm, in case they commit to the bundled product for the next five years.

For the buyer, the decision is clearly not obvious. Maybe people grow tired of the peer feedback tool after three years? Plus, any work you do here would clearly help the business; it might be the last insight needed to lock in this unprecedented deal. So, when the account manager comes to you and asks if you can do a statistical analysis that shows whether strong usage patterns of the tool in the first few quarters indicate likelihood of continued usage in the last six months of a deal, it might be tempting to roll up your sleeves and immediately get into the weeds.

Before you do that, first assess what you already know and have from a data perspective:

• (a) You know that you don’t have any other five-year deals as means of comparison, if you were to perform a statistically sound analysis. Two years is the longest multi-year renewal your company has had so far. That said, it doesn’t mean you can’t extrapolate.
• (b) From previous analyses you and other analytics teammates have done, you know that for the two-year deals, on average, 85% of the customer’s employees request peer feedback in this tool at least once in a 12-month period.
• (c) Of those 85%, a typical customer employee requests peer feedback in this tool on average 1.5 times per month.
• (d) You also know that requests are very frequent in the first and the last quarter of the year, and are less frequent over the summer months in the third quarter.

Then you also want to assess if there are sensible assumptions you can make about the length of this potential 5-year deal. Particularly:

• (e) Five years is a long time. In the modern economy and especially in the tech industry, many employees will not stay at one place for 5 years. If someone joins the customer company at the beginning of this deal, it is likely they will leave the customer company and go elsewhere by the time this multi-year deal is over.
• (f) Company leadership will change in those five years. Employees will switch teams and work with new peers in those five years. All to say, things will keep changing and will not stay the same in that 5-year period. Which means it’s likely they will want to keep hearing feedback from their (new) peers.

Knowing all this, you realize that you might already have a lot of the info you need to give a reasonable recommendation. So, you tell the account manager:

“Thanks so much for your request! I have noodled on this for a bit, and I think we can derive an answer that might be helpful to the customer. Here is how I would break it down:

• (1) It is hard to have a statistically sound analysis around this because we have never had a 5-year renewal deal before. It’s also impractical to project correlations between early and later usage across such a lengthy deal because (as I’m sure the buyer will be also aware) so many things can change in five years! Employees will leave, they will change teams, they will get new managers, and hopefully many other good changes will happen.
• (2) What we do know is that for the two-year deals we’ve sold, 85% of employees use the peer feedback tool at least once in 12 months, and of those 85%, an employee requests peer feedback on average 1.5 times a month.
• (3) We expect that the 5-year deal will follow similar patterns, and that the customer will see lulls during the summer months each year, which is expected.
• (4) The customer can also expect that not all of their employees will stick around for 5 years, so we can assume that we will go from 85% to maybe only 30—40% employees who will have used the tool at least once every year in the 5-year span.
• (5) For that subset of users, we can assume that their average usage might taper off in the fourth and fifth year, to around 1 feedback cycle request every 2 months, but that will also be offset by more frequent usage from new employees who joined after the 5-year deal was signed.”

Tada! Yet again, you saved time and money for the business by not spending days on answering this question. This approach might feel hand-wavy, but think about the other scenario, in which you do a full-blown statistical analysis for the business. Would you arrive to a notably different conclusion? Likely not. Your analysis would probably show something similar, and even if you identified the more nuanced insight that employees who engage with the peer feedback tool frequently early on are also more likely to engage with it later on, you would still face the same conundrum—does it hold four to five years in? Probably not, for all the same reasons you already knew.

Step 3 takeaway: While it might feel unrigorous and uneasy, always ask yourself if you can extract most of the needed insight (“80% of what you need”) with a back-of-the-envelope-calculation (“20% of the work”). If the answer is yes, resist the temptation to do full-scale analytics. You will otherwise be delivering diminishing returns to the business. Save the time and energy for business problems that will truly require the precision and the rigor.

Those are the first three steps! Even if the project has passed all the three checks, we’re still at a “might be worth doing.” In Part 2, I cover the fourth step—the more rigorous step—that helps us make the final call on whether a “might be” should become a “definitely”.

1. Hyper-condensed acronym for “business-to-business (B2B) software-as-a-service (SaaS).” ↩︎