Understanding Statistical Significance with Omniture Test&Target

Already 2010 is feeling like the year of optimization. Everywhere I look, I’m seeing conversations about A|B and MVT testing, optimizing conversion flows, and understanding statistical significance.   When I first started running A|B tests, everything I did was on faith.  I had good intention, I measured all the key indicators, but I had no idea how to tackle the question of “yeah but, is it statistically significant?”  Then I began to crawl as I experimented with online calculators and eventually I moved on to building out my own formulas in Excel but still there was little confidence in myself, let alone the test results.   Eventually I began to experiment with testing tools like Google Web Optimizer, Amadesa, and Omiture Test & Target.  This seemed to make life so much simpler as all the questions I was being asked were answered right in the testing application.  Is it significant? Amadesa says they are 98% confident in the results.  What is the lift we are seeing? Google Web Optimizer says its 8.5% and as a bonus it gives the confidence interval.   While I think it is extremely valuable to have your testing and optimization platform provide the key statistical measures that relate to your test, I think it is just as important to understand the math behind the reports, after all, you can’t call yourself a “car guy” or a “car girl” if you drive on the gauges alone and you don’t understand how the underlying systems work.   Let’s walk through an example campaign to understand how Omniture Test & Target calculates the statistics behind the results....

Understanding Statistical Significance with Omniture Test&Target

Already 2010 is feeling like the year of optimization. Everywhere I look, I’m seeing conversations about A|B and MVT testing, optimizing conversion flows, and understanding statistical significance. When I first started running A|B tests, everything I did was on faith.  I had good intention, I measured all the key indicators, but I had no idea how to tackle the question of “yeah but, is it statistically significant?”  Then I began to crawl as I experimented with online calculators and eventually I moved on to building out my own formulas in Excel but still there was little confidence in myself, let alone the test results. Eventually I began to experiment with testing tools like Google Web Optimizer, Amadesa, and Omiture Test & Target.  This seemed to make life so much simpler as all the questions I was being asked were answered right in the testing application.  Is it significant? Amadesa says they are 98% confident in the results.  What is the lift we are seeing? Google Web Optimizer says its 8.5% and as a bonus it gives the confidence interval. While I think it is extremely valuable to have your testing and optimization platform provide the key statistical measures that relate to your test, I think it is just as important to understand the math behind the reports, after all, you can’t call yourself a “car guy” or a “car girl” if you drive on the gauges alone and you don’t understand how the underlying systems work. Let’s walk through an example campaign to understand how Omniture Test & Target calculates the statistics behind the results. For our campaign, lets...

Validating Page Tags with HttpFox

Whether you are validating the deployment of a new measurement solution, testing enhanced page tracking, or debugging an issue with your site, having quick access to the data your site measurement tool is collecting is a must.   I have used debugger tools provided by the web analytics vendors, I have used network analyzers like Charles, Ethereal, and Wireshark and tools built specifically for web analytics like WASP.   However, I find myself always going back to HttpFox. It’s Clean. It’s simple. It’s so easy to use, after I was introduced to it by my friend Michael Sanders, I went on to train everyone from quality assurance engineers to CEOs on how to use HttpFox to answer the question “what variables are we capturing on this page?”   How do I use HttpFox to validate my web analytics page tags?   1. Add the HttpFox Firefox extension to your browser.   2. Once you have installed the extension and have restarted Firefox, you will notice a new icon in your status bar.   3. Activate HttpFox by clicking the icon in the status bar.   4. Click the Start button and navigate to a page on your site. You will see a long list of records rolling in, don’t worry, in the next step we will talk about searching for the one record that matters.   5. Depending on what measurement solution you are using, the search string will be different. Here is a list of search strings that I use most often: Google Analytics = _utm.gif Yahoo Analytics = a.analytics.yahoo Omniture = /b/ss/   6. Click on the...