Affected: All SDKs
Symptoms
The percentage rollout appears uneven compared to expected distributions. While you might typically reference Data Export databases, the flag evaluations graph in the LaunchDarkly dashboard, or third-party platforms such as Google Analytics or Adobe Analytics, there may be additional sources pertinent to your data collection.
Cause
Flag evaluations graph
When sourcing data from the flag evaluations graph in the LaunchDarkly dashboard, it's crucial to recognize that the displayed information represents the number of evaluations per variation, not the number of contexts per variation. As a context may trigger multiple flag evaluations, the flag evaluations graph is not suitable for observing distribution.
Third-party provider or custom reporting
Accuracy concerns may arise with data sourced from third-party tools or custom reporting methods. To ensure data accuracy:
- Verify the third-party platform does not sample data.
- Check for any over-reporting or under-reporting.
- Confirm that there are no issues in data delivery to your reporting platform.
- Make sure there are no errors in how your reporting platform receives data.
- Ensure your queries count unique users rather than multiple evaluations.
Data Export
For a more precise measurement of user variation, consider using the Data Export feature, ensuring the count reflects unique users only. LaunchDarkly does not keep a history of flag evaluations, and we cannot conclusively tell you why a flag was evaluated the way it was.
Solution
To validate the accuracy of your distribution measurements, start with our percentage rollout distribution tool: Github Repository for Percentage Rollout Distribution Tool
If the tool confirms correct distribution yet issues persist, consider the sample size of your rollout. The SDK's percentage rollout algorithm evaluates flags in real-time using the user's key as a seed to ensure consistent variation delivery—details on this are available here: Understanding Percentage Rollout Logic
Small sample sizes may result in skewed distributions, but these typically normalize at larger scales. If your implementation appears correct and the sample size isn't a factor, the issue might lie within the SDK's execution:
- Utilize the SDK's Evaluation Reasons feature to identify whether it's serving the rollout correctly or encountering errors.
If uncertainties about your percentage rollout distribution remain, please contact support with the following:
- Data indicating a deviation in the rollout.
- Confirmation of accurate data reporting.
- Results from the percentage rollout distribution tool.
- Any SDK errors impacting data.