Answered Different results using hand calculations for A/B testing?

[Melanie]

The past few days I've been having issues with TubeBuddy pulling data for A/B tests and this morning my results came in (woot!)

Anyway, I'm a stats nerd and so I decided to calculate my A/B test results by hand using a Two-sampled t-test and I'm getting different results than TubeBuddy.

According to TubeBuddy, one of my A/B tests has 0% certainty, but according to a rough glance at it there looks to be a difference. When I do it by hand, I'm able to reject the null hypothesis with 95% certainty.

Does TubeBuddy not use a two-sampled t-test?

My Data
Sample one:
mean = 0
σ = 0
count = 5

Sample two:
mean = 0.049
σ = 0.0292
count = 5

 

[ChrisHuff]

TubeBuddy's A/B testing simply alternates the thumbnails each day until it reaches statistical significance (if that's the option you selected). You don't really need a two-sampled test because the sample size itself determines whether it's significant.

But if you're curious as to whether the sampling that made up the two groups caused the results to be biased, you could also ways run the test again to see if you get the same outcome.

 

[ChrisHuff]

TubeBuddy's A/B testing simply alternates the thumbnails each day until it reaches statistical significance (if that's the option you selected). You don't really need a two-sampled test because the sample size itself determines whether it's significant.

But if you're curious as to whether the sampling that made up the two groups caused the results to be biased, you could also ways run the test again to see if you get the same outcome.

 

[Stanley | Team TB]

You should also be aware that portions of these results may be delayed by up to 48 hours. The data/insights you see may reference data that is not yet represented entirely in the stats provided.