A practical guide on Testing and Optimization

  • strict warning: Non-static method view::load() should not be called statically in /home/webpersu/public_html/sites/all/modules/views/views.module on line 879.
  • strict warning: Declaration of views_handler_filter::options_validate() should be compatible with views_handler::options_validate($form, &$form_state) in /home/webpersu/public_html/sites/all/modules/views/handlers/views_handler_filter.inc on line 589.
  • strict warning: Declaration of views_handler_filter::options_submit() should be compatible with views_handler::options_submit($form, &$form_state) in /home/webpersu/public_html/sites/all/modules/views/handlers/views_handler_filter.inc on line 589.
  • strict warning: Declaration of views_plugin_row::options_validate() should be compatible with views_plugin::options_validate(&$form, &$form_state) in /home/webpersu/public_html/sites/all/modules/views/plugins/views_plugin_row.inc on line 135.
  • strict warning: Declaration of views_plugin_row::options_submit() should be compatible with views_plugin::options_submit(&$form, &$form_state) in /home/webpersu/public_html/sites/all/modules/views/plugins/views_plugin_row.inc on line 135.

Testing and Optimization is no longer an exclusive topic in the Web Analytics world.There are some great posts from thought-leaders and I have spent a significant amount of time studying and researching them and their approaches. In my readings I have determined that a good number of posts talk about (or remain limited to) tips and definitions.  There seems to be a lack of proper framework or process around cultivating the testing culture in the enterprise. What are the steps an organization can take to embark on the Testing & Optimization journey? Are there any best practices (real world studies) which one can learn from and potentially apply in their cases?

My goal in this persuasion is to provide quick descriptions as well as real world examples on how an organization can consider approaching Testing and Optimization as a philosophy (and not a project). It will then build on to setting up an iterative testing and optimization process for your organization. A testing and optimization maturity model of sorts.  My belief is that after reading this post you will be able to determine the testing technique you should start with (and scale to).

Think about the last time you tried to do something new; whether it was riding a bike, learning to swim, or pushing the dial by a few pounds while lifting your weights? It was taxing right? At the time it felt impossible. You had no idea whether you'd be able to get through. But you did it and you survived! In most instances, anytime you try to something new or venture into an unkown territory, you will find it to be mentally challenging.

Don't give up though. Just keep pushing yourself.  Once you start, understand the basic concepts, and get disciplined you will find it to be self-fulfilling, addictive, and in this case one of the easiest and funnest aspects of web analytics.   Remember, digital is the most measurable medium in the history of marketing; With testing you have the opportunity to leverage it to its fullest.

So let's get the basics out first:

What is Multivariate Testing (MVT)?

This form of testing enables you to test more than one component of your website. You can test display ads, e-mails. or landing pages in the same live environment.  When used correctly this form of testing delivers significant improvements quickly!

What output should I expect from leveraging MVT?

It usually determines which content or creative (or a combination of them) produces the best improvement in the defined goals of the test.

What are some of the tips to keep in mind when conducting MVT?

A golden (and first) rule around experimenting is hypothesis generation. A hypothesis must link a test outcome to user-behavior on the site. Once the testing hypothesis is created it's also important to establish metrics. Consider selecting complex metrics or derived metrics (KPIs) as opposed to simple ones (e.g. click-throughs and conversion rate).

What is A/B Testing?

This involves testing one component of a website at a time in a live enviornment. It is also referred to as "split testing" because the Digital Traffic is divided (split) per options tested. This form of testing is usually a simpler alternative to MVT.

For most organizations, this is generally a good place to start. If you are a  beginner or have a  low performing website, I highly recommend using this approach to quickly identify and hypothesize on the low hanging fruit(s) and building up from there.

What output should you expect from leveraging A/B Testing?

It is a great way to determine which single content or creative variation (from the options tested) produces the best improvement based on the hypothesis and defined goals of the test. A/B testing is a good approach to philospohically align the organization on an iterative Testing and Optimization schedule.

Sidebar on testing in general: As a practitioner, I know this can be a difficult sell particularly if there is lack of digital savy. I have also heard a left brain right brain struggle in this process. (e.g. let bygones be bygones and focus on the next best expereince). If you face these pushbacks and challenges I encourage you to surface three things: (a) power of learning from mistakes (b) keeping your analysis rock solid particularly on the basis of opportunity cost and (c) never giving up.

The most persistent always wins. So,  keep trying, don't give up and focus your analysis on economic value.

What's the difference between A/B Test and MVT?

The main difference between the two is the ability to test one feature at a time vs. multiple features simultaneously. A/B testing is useful when you don't have the basics right on your site or if you are just starting out.

What is Usability Testing?

This last form of testing involves collecting data or information through one-on-on observations via focus groups and/or remote testing. It is an experimental form of testing (not market research based) as it involves asking real people about the site experiences WHILE they are using your website. I personally am a firm believer in using this form of testing early in the process. Usability testing combined with Web Analytics can be an extremely powerful mechanism to fine-tune or optimize your site.

What output should you expect from leveraging Usability Testing?

I have a personal bias towards this form of testing. I like it because this form of testing can provide instant feedback on optimization areas. To be able to get a real world, first hand perspective from an average web visitor  in real time is nothing less than magical.

Be prepared to be challenged by your ego. But if you can endure it it's the best thing you can do as a good (true) marketer.

Sidebar on Usability Testing: As a practitioner I often get questions on the number of observations or samples required for good usability tests. For any experiment it's good to have a large sample size to provide statistically significant data, however, with a handful of observations one can still generalize barriers to conversion (or goals) with this exercise. Net, net whether you make 3 observations or 300 this form of testing is worth every penny and must be a part of your Testing strategy. The key here is to focus on a process or a section of the site vis-a-vis competitor (if possible) or from the lens of the average user.

Here is a quick cheat sheet  (testing summary matrix) for your reference as you embark in this wonderful journey. Click on the Image to expand or e-mail me to secure a soft copy .

Testing and Optimization MatrixTesting and Optimization Matrix

Now that the basics are covered and you have managed to steeer the organization into embarking on a testng journey, what next?

Focus on results.

How to write case studies for your tests?

I have created an example of a real life case study performed around A/B/n testing. A very important part of being in Web Analytics as a leader or practitioner is to be able to position yourself by selling or advocating your work. Action speaks louder than words,  so the let the results of your actions speak for themselves. 

Testing and Optimization Case StudyTesting and Optimization Case Study

I also intend to refresh this specific post regularly by adding  interesting studies as time goes on. You can also e-mail me if you would like to receive soft copy versions of other case studies as I secure them.

Stop thinking and Start Testing!