A / B Testing Revisions

 Session #1

Notes

#A/BTesting = #Randomizedcontrolledtrials of two versions of same application

• Running experiments
• Used in Testing search, pricing algos
• Origins of A/B testing 100 years back
• Testing two fertilizers
• Commit to sample size, Number of pots to use
• More pots, Higher sample size honest assessments
• Analyze by hypothesis test
• The difference between two options

• Binary classification false positives
• What might have happened if no difference

• Current state
• Adjust test lengths in realtime
• Adjust duration of test

• How long test running
• Variations of page
• Views / Example clicks
• Compared to baseline comparison
• A-A Testing both variations are same
• 5% False positive probability
• Number of experiments, Sample size, 50% of actual size of customerbase
• Optimal procedure - Monitoring and Stopping early

• Statistical significance of confidence level

• Staging trials
• Wait to see p value < 5
• Secondary diagnostics
• Sample size calculation
• How many samples to wait for

Session #2

• Conversions / clicks measure
• Prior Beleif + Update prior belif and Probability distribution

Sample sizes

• Some marketing experts even recommend sample sizes of up to 5,000 people.
• One way is to run A/B tests separately for specific devices and browsers.
• statistical significance of 95%

P > 0.05 is the probability that the null hypothesis is true

P ≤ 0.05 means that the test hypothesis is false or should be rejected

A/B test sample size calculator

How to Calculate A/B Testing Sample Sizes?

On the Complexity of A/B Testing

An architecture for enabling A/B experiments in automotive embedded software

Offline A/B testing for Recommender Systems

Keep Exploring!!!!