A/B Testing

What is A/B testing? It is conducting experiment to optimize customer experience. If the test is a success, we will then use the winning test to influence the behavior of the customer to our benefit (the one defined in the test).

An A/B has 4 steps:
1) Analyse the data
2) Form an hypothesis
3) Construct an experiment
4) Interpret the result

1) Analyse the data

Quantitative data tells where to test (get this data by behavior analysis...).
Qualitative data tells what to test (get this data via survey, feedback...).

2) Form an hypothesis

"If [variable], then [result] because [rational]."
[variable] = the element that is modified:

• Call to action wording, size, color and placement
• Headline of product description
• Form length and types of fields
• Layout and style of website
• Product pricing and promotional offers
• Images on landing and product pages
• Amount of text on the page (short vs. long)

[result] = predicted outcome derived from the analysis of data is 1) (more sign-up, improved CTA, % increase, number goal etc...)
[rational] = what will be proven wrong if the test fails?

ex:
"If the navigation band is removed from the check-out page, then the conversion rate at each step will increase because our website analytics shows that traffic drops due to click on these links."

3) Experimentation

Every test have 3 parts:

• Content: what does it says?
• Design: how does it look?
• Tech: how does it work?

4) Interpret the result

We want to answer the question: How confident are we that the results are not due to chance?

Mathematically, the conversion rate is represented by a binomial random variable, which is a fancy way of saying that it can have two possible values: conversion or non-conversion.

(The binomial distribution with parameters n and p is the discrete probability distribution of the number of successes in a sequence of n independent yes/no experiments, each of which yields success with probability p.)

To declare a winning we want the result of the test to be statistically significant at 95% which mean that only 5% probability that the result was due to chance.

This equate to having all the tested Conversion rate within 1.96 SE from the average conversion rate C.

To avoid doing repeated experiments, statistics has a neat trick in its toolbox. There is a concept called standard error, which tells how much deviation from average conversion rate (C) can be expected if this experiment is repeated multiple times. Smaller the deviation, more confident you can be about estimating true conversion rate.

For a given conversion rate (C) and number of trials (n), standard error is calculated as:

Standard Error (SE) = Square root of (C * (1-C) / n)

you can be sure with 95% confidence that your true conversion rate lies within this range: C % ± 2 * SE
(1.96 to be exact).

Testing the statistical significance corresponds to


p is the conversion rate that we are observing.

GOING INTO THE NITY DETAILS -- NEED CLEANING AND CLARIFICATION
http://www.slideshare.net/dj4b1n/ab-testing-pitfalls-and-lessons-learned-at-spotify-31935130

A/B Testing Pitfalls and Lessons Learned at Spotify from dj4b1n

The likelihood of obtaining a certain value under a given distribution is measured by its p-value 

If there is a low likelihood that a change is due to chance alone, we call our results statistically significant

Statistical significance is measure by alpha which typical has a value of 5% or 1%.
Alternatively P(significant) = 0.05 or 0.01
There is a conversion from alpha to z-Score.
The Z-score tells us how far a particular value is from the mean (and what the corresponding likelihood is)

Standard deviation (σ) tells us how spread out the numbers are

Test of statistical significance:

Null hypothesis is that the conversion rate of the control is no less than the conversion rate of the experiment:

H0: CR - CRc =<0

CRc is the conversion rate of the control.

Alternative hypothesis is that the experimental page has a higher conversion rate. This is what we want to test.

Conversion rate are normally distributed (conversion = success).

What we are looking for is if the difference between the two conversion rate is large enough to conclude that the treatment altered the behavior.

Random variable for the test will be: X = CR - CRc

Null hypothesis: H0: p-Value =<0

We use the Z-score to derive the p-value:

Z-Score = (CRc - Cre)/√(σc^2+σe^2)

p-Value = NORMDIST(Z-Score, 0, 1, TRUE)

if p-Value >95% then we can reject Null hypothesis and the A/B test is good to go!