• Guest - Earn a FREE TubeBuddy Upgrade for being active on the forums! Click Here to learn how you earn free upgrades for TubeBuddy!
  • Guest - TubeBuddy has a discord! Click Here to join in the conversation!

Question A/B Split Testing Question

Mike O'Pray

New Member
2
1
Subscriber Goal
10000000
Hey everyone,

This is my first post in the Tubebuddy forum, be nice ;)

I have a question about A/B split testing, as it seems to have a little bit of a flaw (maybe I'm wrong).

I have attached a few images showing some recent tests.

Tubebuddy states that for all of these tests the "Original will be better than the Variation".

Yes, based on CTR alone, that is true, but is this the only factor we should take into consideration?

If we look at the Impressions/Clicks column, we can see that for all of these tests, despite having a lower CTR, we actually have more impressions/clicks.

To me this would mean that the Variation is actually performing better.

Tubebuddy will automatically select the Original unless I override it.

Please let me know your opinions on this.

Thank you,

Michael
 

Attachments

Stanley Orchard

Amazingly Decent and Not-At-All Terrible Fishing
Administrator
Moderator
TubeBuddy Staff
1,864
24
Subscriber Goal
250000
First off welcome and thank you! For a first post this is an awesome topic!

When you first hear about A/B tests the initial idea is that you run the test and one will get a better CTR than the other... and as you pointed out that is simply not the case. This is the real beauty of the A/B Testing feature; seeing the effect your thumbnail and title have on your video. It is not uncommon to have tests where one thumbnail may get more clicks... but the other gets more watch time. We try not to judge for you with data like this; we are here to provide data for you to make educated decisions but we also want to make things as easy for you as possible. Be mindful of this data when making the settings for your A/B tests; if you set TubeBuddy to choose a particular thumbnail after 'Statistical Significance' you are going to get the results based on CTR and not watch time.
 
OP
OP
Mike O'Pray

Mike O'Pray

New Member
2
1
Subscriber Goal
10000000
Hi Stanley, thanks for your reply. In these tests, I will select the variation in order to continue getting more views and watch time despite a lower CTR, would you agree this is a good idea? To me it seems like common sense, but I just wanted to check I wasn't missing something.
 

Stanley Orchard

Amazingly Decent and Not-At-All Terrible Fishing
Administrator
Moderator
TubeBuddy Staff
1,864
24
Subscriber Goal
250000
Don't judge based on impressions, at least not initially. That metric is there more for comparison sake... unless you have waited +/1 2 weeks after publishing your video to start your test. If you started your AB test the day you published then the impressions metric will be skewed due to the notifications to your subscribers and Home Page/Subscription feeds.

In the cases above I would go with the original in all circumstances because the CTR and the watch time per impression are higher. If there was a discrepancy I would have to spend a little time figuring out what to do. For example, if the videos had a 9.0% CTR and a 10% CTR but the first had an average watch time-per-impression of 37 seconds and the second had 29 seconds I would very easily choose the first thumbnail. That difference in watch time is well worth losing a few actual clicks.

That having been said the single most important thing to do with these tests now is to look at these and see what is similar about them; what makes the originals better than the variations and can you reproduce/improve this?
 

Zanet

Recognized Member
TubeBuddy Legend
99
12
I agree as much as I love the A B testing tool, ultimately decisions heed to factor watch time, and engagement