Question A/B Split Testing Question

[Mike O'Pray]

Hey everyone,

This is my first post in the Tubebuddy forum, be nice

I have a question about A/B split testing, as it seems to have a little bit of a flaw (maybe I'm wrong).

I have attached a few images showing some recent tests.

Tubebuddy states that for all of these tests the "Original will be better than the Variation".

Yes, based on CTR alone, that is true, but is this the only factor we should take into consideration?

If we look at the Impressions/Clicks column, we can see that for all of these tests, despite having a lower CTR, we actually have more impressions/clicks.

To me this would mean that the Variation is actually performing better.

Tubebuddy will automatically select the Original unless I override it.

Please let me know your opinions on this.

Thank you,

Michael

 

[Stanley | Team TB]

First off welcome and thank you! For a first post this is an awesome topic!

When you first hear about A/B tests the initial idea is that you run the test and one will get a better CTR than the other... and as you pointed out that is simply not the case. This is the real beauty of the A/B Testing feature; seeing the effect your thumbnail and title have on your video. It is not uncommon to have tests where one thumbnail may get more clicks... but the other gets more watch time. We try not to judge for you with data like this; we are here to provide data for you to make educated decisions but we also want to make things as easy for you as possible. Be mindful of this data when making the settings for your A/B tests; if you set TubeBuddy to choose a particular thumbnail after 'Statistical Significance' you are going to get the results based on CTR and not watch time.

 

[Mike O'Pray]

Hi Stanley, thanks for your reply. In these tests, I will select the variation in order to continue getting more views and watch time despite a lower CTR, would you agree this is a good idea? To me it seems like common sense, but I just wanted to check I wasn't missing something.

 

[Stanley | Team TB]

Don't judge based on impressions, at least not initially. That metric is there more for comparison sake... unless you have waited +/1 2 weeks after publishing your video to start your test. If you started your AB test the day you published then the impressions metric will be skewed due to the notifications to your subscribers and Home Page/Subscription feeds.

In the cases above I would go with the original in all circumstances because the CTR and the watch time per impression are higher. If there was a discrepancy I would have to spend a little time figuring out what to do. For example, if the videos had a 9.0% CTR and a 10% CTR but the first had an average watch time-per-impression of 37 seconds and the second had 29 seconds I would very easily choose the first thumbnail. That difference in watch time is well worth losing a few actual clicks.

That having been said the single most important thing to do with these tests now is to look at these and see what is similar about them; what makes the originals better than the variations and can you reproduce/improve this?

 

[Muzafir]

cool information....

 

[RobBalasabas]

Welcome to the community, @Mike O'Pray! Epic first post