Eric J. Hansen: So you clearly have a lot of testing under your belt, and I'm sure you expect to be testing more and more in perpetuity. But what sort of advice or anecdotes would you convey to folks that are really just getting started, or certainly don't have the breadth of experience that Vegas.com has?

Mike Brown: One thing that could be an initial obstacle is who should you test to and how long should a test run. There's this concept that a test needs to run until X number of people have gotten through it to be statistically valid, which is all true, but there's other things that can mess up a test if you don't do them right. So one of the things that we always do now is we only show tests to new users. If there's a test on our home page, we can get statistical validity in a day, but that's not going to take into account daily or weekly changes, or seasonality. So we never run anything less than a week or two, and sometimes, that may be too short.

So the advice I would give to people is to test new people. And when you end the test, stop new people from coming into it, but then let the test keep coming. It's like if you were playing baseball. If you had existing people who had been to your site yesterday, it's like changing the strike zone as they're up there. They're like, "Whoa. This looks really different. I don't know what to do." So that makes the test invalid.

And then, if you end it too soon, it would be like saying, "OK. Strike one. You're out." Just because they didn't convert on the first visit doesn't mean that it's a bad test. Most people that come to our website take two or three visits before they've decided and picked what they wanted, and they've asked their wife's permission. So don't call them out. Don't call failure a success until users have had time to get through their purchase.

Eric: So you're actually taking advantage of some of SiteSpect's targeting capabilities. In other words, let's not run these tests against everyone and anyone, but let's target people based on what we know about them as they come into the site: this a new user; this is an existing user.

And then, some of the more fine-grain control that you get within the test campaign of SiteSpect, let's turn it off for new users, but let people who are already in the test, that might come back to the site, stick to the test and measure their behavior, not just on a visit basis, but across visits. So we can look at how people behave from a unique user perspective as opposed to just a session basis.

Mike: Yeah. That may not be basic advice, but it's critical. And also use your segmentation capability. After the test, did everybody respond equally? Or "Oh gosh. This variation won because this variation over here didn't work in Firefox," and that's an invalid test then. There was a tech problem that nullified it, not good or bad design. That's another key thing is looking at your results and your user segments.

Eric: This is a lot of good information, particularly for people that might be getting their feet wet with testing, or maybe they've got some experience, and they really want to kick it up to the next level. Now based on our research, and what we read from other research firms, the number of sites out there that are doing testing and optimization is still very low, less than 10 percent.

So clearly, there's a lot of people that could benefit from testing and optimization methodologies, but they're just not doing it for whatever reason. What would you say to these people to help them take that plunge and run a first test, or try to convince their boss this is something that's worth spending some time on?

Mike: My advice for them would be to not call you. I would call Google. I would take their free Google Optimizer and try it out. Whose boss needs to be convinced to do something for free? Not many. As long as it doesn't break the site, go do it. So go try out the free tool. It's great for people to get the concepts. I'm told you quickly hit the wall on it. I've never tried it, but it's free. You get what you pay for. And when you're ready to move on, then give you a call. And give your customers a call, or Offermatica, or other people's customers. Talk to real users, because these salespeople...

Eric: You've got to do your research. Absolutely. We find that the best customers are the ones that go through that effort. They do know exactly what they're getting, and why the product is different for their particular needs and maybe the best thing. So here's one more question for you.

As they say in Vegas, this is the money question. For the folks who are testing and have plenty of wins under their belt, and they've had a lot of positive experiences, and they've learned a lot, but they're not using SiteSpect. What would you say to these folks? What can they learn that they might not already know about the vendors of the tools?

Mike: I have to talk about my own needs. My own needs are to be able to test fast and quick, and to not be dependent on everybody else's resources because there are not a lot of those.

If those people are not using SiteSpect, but are doing testing, and are in a similar situation, and they're feeling some constraints of having more ideas to test then they have IT bandwidth to do it with — then I would say this really, really isn't a factor for me. If you like doing one or two tests a month then stay where you're at. But if there's value in being able to do tests every day, multiple tests always, then you may have the wrong solution.

Eric: Very good advice. I definitely appreciate your time and your wisdom and sharing that with our viewers. I'm Eric Hansen from SiteSpect with Mike Brown from Vegas.com. Thanks for watching.

Back to Part 1 of the interview transcript