How to Choose the Right AB Testing Tool: People and Process Matter

Recently, I had the distinct pleasure of guiding a client through the Request for Proposal (RFP) process to help them select an appropriate AB testing tool (referred to as the “tool” from here on out). While I had a fairly good idea of what to expect going in, I was pleasantly reminded about how important people are in the selection process. When it comes down to it, most tools can fulfill the basic requirements of an organization’s testing program.

With that being said, What's the differentiator, then? What truly sets one tool apart from the rest? 

In my experience, it’s two things: (1) a scoring framework to objectively compare tools and shortlist a few and (2) the team of people behind the tool itself.

The Framework

The scoring framework needs to account for the following:

(1) requirements needed in the tool to meet your specific business needs

(2) current industry trends with tools (AI for example)

(3) any known issues with the tool and company (support for example)

(4) functionality and services that are considered “add-on”

Here is a framework that a client and I used recently to help assess and identify potential tools. This one is client-specific, so you will want to think of other attributes to add or remove from this framework. You may not have a CDP or an analytics tool for example.

Again, it all depends upon your specific needs and your current marketing technology infrastructure.

33 STICKS TOOL ASSESSMENT FRAMEWORK

Here are some ways to think through these attributes as you develop your own scoring framework for your company

CDP Integration 

You may need the tool to integrate with your existing CDP. Some tools can handle this well and accommodate data pushes and pulls allowing for not only the enriching of the behavioral data of your site users but also laying the framework for true personalization on the site.

Other tools, however, may not be able to do either of these or maybe just one of these – sending behavioral data sets and audiences to the testing tool.

Ask the vendor about any potential future plans and or partnerships that they may have in the works with your particular CDP.

Analytics Integration 

You also may need the tool to integrate with your analytics platform. Again, depending upon the tool, this integration could be two-way, one-way or not available at all.

An analytics integration accomplishes a few things. First, it maintains a “one source of data” reporting for the business that your key internal stakeholders are used to seeing. This may or may not be important to your business, but it is something to put some time in thinking about when selecting a tool.

Secondly, some tools have amazing reporting capabilities within the tool itself. Data can be monitored, segmented, analyzed for insights and shared quite easily with senior leadership.  However, there are tools that lack functionality that allows for deep dive analysis and insights within the tool itself. In that case, you want to push your testing data into your analytics tool to perform that analysis and generate those insights.

Add-Ons

This is where it can get very complicated and confusing. Ask once and ask again – what do I get under my basic contract and what is considered add on and additional cost?

Add-ons include functionality such as product recommendations, AI functionality, and even customer support!! Make sure you understand what your support would look like once the tool is in place.

Negotiate

A few final notes here. Many vendors do not charge a separate fee for the initial set up of the testing tool. However, a few do charge. Make sure to capture this information as you progress through your selection process.

Also, some vendors put their best pricing forward up front. Others do not. There is always room to negotiate a better price.

Many vendors use unique daily visitors to assess contract costs. Again, others do not.

Finally, go with a one year contract at first, unless of course the vendor is offering significant pricing discounts to extend to a multi-year contract.

 The People

The second facet of selecting a tool is centered around the people that are employed by the tool’s vendor. While it is important to separate the tool from the people when assessing functionality and meeting requirements as discussed above, it would be foolish to ignore the people aspect of this process.

Here’s why:

Some vendors really understand that listening to YOUR NEEDS is the most important part of the process. It is very easy to tell when this happens.

Other vendors may listen to your needs but end up placing THEIR NEEDS in front of your needs. This one is a little trickier to spot.

Finally, some vendors sadly will IGNORE your needs and will easily sell you what you don’t need and shouldn’t buy. Again, this one is easy to spot.

Here are some things to listen for during your process.

• Does the Vendor Understand Where Your Company is in the Testing Maturity Curve?

• Are They Helping Position You for Success?

• Are They Trying to Upsell Too Hard?

• Are They Pushing for Longer Term Contracts When it Doesn’t Make Sense?

• Do They Go the EXTRA MILE During the Process?

• Do They Honor Your Process – This One is HUGE!

• What Does Your Gut Tell You About Them?

As always, I enjoy creating this content for our industry. We need more it quite frankly.

I hope it has been helpful to you, and that you may be able to use this information to make a positive impact at work and with your customers.

Happy Working!

-Jason B