A/B Testing

How Spara supports A/B and multivariate testing of agent behavior.

Spara provides support for testing multiple agents at once. This can be used to test new behavior without having a major impact on your funnel performance.

Anything that is unique to an agent can be tested. As a rule of thumb, this includes anything in the agent's Agent Editor page:

  • AI instructions (prompt)

  • Preview messages and "nudges" (coming soon)

How to A/B test multiple agents

Contact your CSM to set up an A/B or multivariate test. We are working to make multivariate testing functionality self-service soon.

Each agent has starting criteria. See Configuring Agents if you need a refresher.

To test multiple agents, simply define starting criteria that will route a percentage of your buyers to each agent. Let's use an A/B test of Chat agents as an example:

  • Build two Chat agents...

    • "Experiment" chat agent is prompted to drive visitors to the Sign Up page.

    • "Control" chat agent is prompted to drive visitors to schedule a call with your sales team.

  • Configure "Experiment" to activate for X% of website traffic. All other traffic will engage with the "Control" agent.

You can then analyze the results on the Analytics page. Use the Agent filter to compare metrics and outcomes between the two agents.

How to use a third-party analytics tool with Spara

Spara does not support third-party analytics tools. The following guide outlines how a customer could leverage a third-party tool to power Spara A/B tests.

Customers may use Spara's Web API or JavaScript API to set any fact for a lead - which could be used to power an A/B test within Spara.

For example, a customer could...

  • Set up an A/B test in Amplitude, putting users into either an "Experiment" or "Control" group.

  • Customer loads a Amplitude on their website. Receives a callback from Amplitude with the UUID and experiment/control fact for each website visitor.

  • Customer uses Spara JavaScript API to set Amplitude UUID & experiment/control fact for each website visitor

  • Spara automatically sorts website visitors to one of two Spara Chat agents: Experiment agent or Control agent.

Note that this must be done in manner that does not break the buyer's experience. For instance, if a buyer is halfway through a conversation with the Spara Chat Control agent the customer should not update website visitor to engage with the Spara Chat Experiment agent.

How to load AI Chat for a fraction of website visitors

A common use case during onboarding is to only load Spara for a small percentage of website visitors.

Replace your Spara javascript embed with the following code. This segments website visitors into one of two buckets: a) your website loads Spara, or b) your website does not load Spara. This resets after 30 days, so that returning visitors are not permanently in one bucket or the other.

Set SPARA_TRAFFIC_PERCENTAGE to tune how much of your web traffic sees Spara loaded.

<script>
var SPARA_TRAFFIC_PERCENTAGE = 10; // Set your percentage here (10 = 10%, 100 = always show)

(function() {
  var k = 'spara_traffic', d = 30;
  var s = localStorage.getItem(k);
  var run = SPARA_TRAFFIC_PERCENTAGE >= 100 || (s ? JSON.parse(s).i && Date.now() < JSON.parse(s).e : Math.random() * 100 < SPARA_TRAFFIC_PERCENTAGE);
  
  if (!s || Date.now() > JSON.parse(s).e) {
    localStorage.setItem(k, JSON.stringify({i: run, e: Date.now() + d * 864e5}));
  }
  
  if (run) {
    var script = document.createElement('script');
    script.src = 'https://app.spara.co/embed-<app_id>.js';
    document.head.appendChild(script);
  }
})();
</script>

Note that it is usually easier to accomplish similar behavior with the A/B Testing approach:

  • Load Spara's Javascript snippet for 100% of website visitors

  • Build two Chat agents...

    • "Experiment" chat agent pops up a preview message.

    • "Control" chat agent does not, i.e. it only shows the avatar (bottom right circle)

  • Configure "Experiment" to activate for only X% of website traffic. All other traffic will see the "Control" agent.

Last updated