Whenever you’re split-testing factors in a pay per click advertising account, it’s easy to slack off and skip a rigorous analysis of your results. Option A got a higher conversion rate, so it must be better than Option B, right? Well, sometimes you just have to do the statistical legwork to verify your results. It’s easier than you think.
First, a quick primer on statistics. Whenever you create a research project testing two factors (like two different text ads, or two different match types for the same keyword), you’re bound to get different results for each factor by the end of the test. But you need to know whether this difference was caused by the factors actually being different, or if it just happened by random chance. Here’s where statistical testing comes in. Calculating the difference between two groups can be easily done by performing a “z-test.” You can get all the boring details at Wikipedia’s page on z-tests, or if you’re more interested in the result than the process, you can find a lot of online calculators that will do all the hard work for you. You can find a really good one at the Dimension Research z-test calculator here.
Calculating a z-test is easy. You just punch in your sample group size from your first group (in most cases, you’ll use the total number of clicks on your test factor), type in your frequency or percentage (number of conversions or conversion rate), then repeat the process for your second factor. Hit calculate, and you’ll get a result called your “confidence level.” The confidence level is the statistical chance that the result you are testing did not happen by random chance alone. Therefore, if you get a 95% confidence level on a z-test comparing Text Ad 1 to Text Ad 2, you’re 95% sure that one of the ads is better than the other, and it did not happen randomly. You’ll want to shoot for a 95% confidence level, since this is the acceptable level of confidence for most academic statistical tests.
The Dimension Research calculator also gives you confidence levels for one-tailed and two-tailed tests. Use the one-tailed confidence level if you’re trying to test if one factor is better than the other, and use the two-tailed level if you’re trying to test that the two factors are equal.
After running a few statistical tests, you’ll probably find that a lot of split-tests that initially looked significant aren’t very significant at all. By adding statistical rigor to your marketing tests, you can ensure that your analysis is accurate. You won’t make mistakes in judgment if you trust the numbers.