How can we help?

How are A/B Test Results Calculated?


When a conversion event comes in, we try to find the 'closest' feature event. To do this, we look for the latest feature event for this user in the 24 hours preceding the conversion event. (Failing that, we will look for the earliest feature event for that user in the 5 minutes following the conversion event, to allow for clock skew). Once we find a feature event, we look at which variation that user saw, and we add the user to the set of users who saw that variation and converted.

So, suppose the following timeline:

t0: user X sees variation A
t1: user X sees variation B
t2: user X triggers a conversion event

As long as there is less than 24 hours between t2 and t1, user X will be added to the variation B conversion set (and *not* the variation A conversion set).

In this same timeline, but when there is more than 24 hours between t2 and t1, we will not find a feature event, so the conversion will not count at all.

Suppose an alternate timeline:

t0: user Y sees variation A
t1: user Y triggers a conversion event
t2: user Y sees variation B
t3: user Y triggers a conversion event

If there is less than 24 hours between t1 and t0 AND less than 24 hours between t3 and t2, then user Y will be counted in both the variation A conversion set *and* the variation B conversion set. 

Now, suppose a third timeline:

t0: user Z sees variation A
t1: user Z triggers a conversion event
t2: user Z sees variation A
t3: user Z triggers a conversion event

If there is less than 24 hours between t1 and t0 AND less than 24 hours between t3 and t2, then user Z will be counted *once* in the variation A conversion set.

Keep in mind that the important time limit here is 24 hours. If someone converts 5 days (or 2 days) after (or 2 days) getting a variation, they will not be counted.

Contact us at support@launchdarkly.com if you have any additional questions.

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request