📣 Tests every Product Manager should know 📣
The joy of building a product is one thing and shipping it to customer to use is another. With Agile Methodology in practice, the products are shipped in iterations and requires continuous improvements throughout its lifecycle.
Considering the iterative nature of products it is indispensable to validate assumptions and lay a viewpoint about what experience is better for the user.
Let’s go through few tests that Product Managers (PM) should know, in order to make effective decisions.
We talked about validation above that is usually done through Hypothesis Testing. During this testing we outline a hypothesis for a change and then define the success. To account for changes, it’s critical to isolate different variables in the product domain. Along with that we need to be careful of satisfying the assumptions that are associated with hypothesis testing.
WHAT IS HYPOTHESIS TESTING 🤷🏻♀️
Hypothesis tests are used to assess and understand the plausibility or likelihood of some assumed viewpoint called as hypothesis. This hypothesis is based on data.
Let’s say we have 2 different free trial features within our product and we find ourselves in dilemma :
Feature 1 — Subscribed users = 12 | Conversion ratio : 60%
Feature 2 — Subscribed users = 101 | Conversion ratio : 48%
The feature 1 seems to be a better performer in Conversion ratio but has been subscribed only by 12 users.
Every PM shall say : I need more confidence before making a conclusion. LET’S TEST IT !!!
There are some mathematical terminology that you have to understand before we can set up the testing framework :
- Null Hypothesis -It is the commonly accepted fact. The null hypothesis tends to state that we have a viewpoint and there’s no change in it.
- Alternative Hypothesis -It is usually what we want to test. It is opposite of null hypothesis
- Significance Level (alpha) -It is probability of rejecting the null hypothesis
- P Value -It is probability of getting the test results. The smaller value implies we go with alternative hypothesis
- Acceptance Criteria -It is the threshold we set to determine if there is enough evidence to support the null hypothesis. This is often set to (standard) p value = 0.05 but if we want more certainty we can lower this value.
Note: Probability is a statistical and mathematical concept; which simply means how likely is something to happen.
COMMON TESTS
🔶 One Sample T-Test
This test looks to assess differences between a sample and the entire population from which that sample is derived.
Example: Is the average monthly revenue among my customers significantly more than the SaaS Subscription industry this year?
🔶 Two Sample Independent T-Test
This test looks to assess difference between one sample and another sample.
Example: Does the feature A used by one set of customers has better conversion rate than feature B used by another one set of customers (This is also know by A/B Testing in industry)
Example: Is my churn rate on and average significantly less than our close competitor ?
🔶 Paired Sample T-Test
This test looks to assess differences between sample and that same sample but at another point in time.
Example: Did our customer’s stickiness with our product increasing due to introduction of different pricing levels over period of one year ?
🔶 Chi-square Of Independence
This test is used to determine if there is a relationship between two categorical variables. This is done by calculating the actual and expected frequencies to look for dependence between two variables.
Example: Are the newly introduced in app feedback forms response percentage (%) higher than of the traditional emailed feedback forms.
SETTING UP TESTING FRAMEWORK 📝
Now that we know about these tests, we can set up the test design, as follows :
🥇Define hypothesis to be tested : To investigate the situation, setting up the hypothesis well is critical. This is the first step in designing the test
🥈Decision making criteria for the chosen test : The test we select shall validate our hypothesis. There are many statistical and mathematical criteria which can be used to infer conclusions
🥉Success metrics for the test : In the product space, we shall have a quantifiable measurement that we can track. They are also known in industry by another name is Key Performing Indicators (KPI)
KEY POINTS TO KNOW
🧩 Hypothesis test comes with their own set of assumptions. To ensure test give us the correct results we shall always look at whether the assumptions where satisfied or not and then only proceed further in the process
🧩 The test runs on samples and to ensure we are not introducing any bias in sampling, we need to follow correct sampling procedures and methods. (I have an article on this 👉🏼 Biased Samples )
🪶 Want to know more about the T test ‣ Read this
🪶 Read more on different types of T test
🪶 Read about A/B Testing
⦿ References : The Whole Product Community ⦿
👋🏼 👋🏼 Until next time 👋🏼 👋🏼
