Network troubleshooting checklist

Overview

Use this document to troubleshoot unexpected flag evaluations or errors in the streaming or polling connection. Follow the steps in chronological order to pinpoint the issue. At any point, if you need assistance, contact Technical Support.

This checklist applies to all LaunchDarkly SDK and Relay Proxy connection issues. For a list of LaunchDarkly SDKs, refer to the LaunchDarkly documentation.

Solution

Part 1: Identify

1. Are you seeing a 5xx error from LaunchDarkly endpoints?

   • Yes: Contact Technical Support.

2. Is this problem occurring in a brand-new installation of a LaunchDarkly SDK in an existing process/application, where LaunchDarkly hasn't been previously used or confirmed working?

   • Yes: Go to Initialization Troubleshooting.
   • No: Did the process or application restart or open a new instance when the issue began?
     • Yes: Go to Initialization Troubleshooting.
     • No: Go to Connection Interruption Troubleshooting.

3. Did the error occur on the SDK initialization, i.e., when the SDK first attempts to create a connection to LaunchDarkly or Relay Proxy streaming or polling endpoints?

   • Yes: Go to Initialization Troubleshooting.
   • No: If the error happens after a successful connection to the service for any length of time, go to Connection Interruption Troubleshooting.

4. Do you receive the same error using a command-line tool to interact with our streaming/evaluation APIs? ~ Review, Check the network connectivity through the appropriate cURL (or the Windows equivalent), and see if you encounter the same error in your logs.

   • Yes: Skip to Step 6.

5. Can you reproduce the issue with a LaunchDarkly sample app in the same language/framework? ~ Locate the relevant SDK documentation to find a link to the LaunchDarkly sample "Hello World" app.

   • Yes: Contact Technical Support.
   • No: Go to Initialization Troubleshooting.

6. Is the application using the SDK or Relay Proxy in offline mode?

   • Yes: Review the Offline Mode Documentation. If the issue persists, contact Technical Support.

7. Does the error only occur when sending events, i.e., is the error only on the endpoint https://events.launchdarkly.com or https://mobile.launchdarkly.com?

   • Yes: Review Troubleshooting no data in live events.

Server-side SDKs

1. Is the SDK running in daemon mode (connecting to a persistent store like Redis, DynamoDB, or Consul)?

   • Yes: Contact Technical Support.

Client-side SDKs

1. Is the issue affecting all users or only some?

   • If only some, check their network connectivity.
   • If users are in a locked-down environment, ensure firewalls allow connections to LaunchDarkly.

Part 2: Investigate

Initialization Troubleshooting

1. Are your logs showing a 401 HTTP response?

   • Yes: this means that the SDK key required to authenticate the connection to your LaunchDarkly environment is not present or is otherwise incorrect. Check your environment variables or the location where this key is stored and retrieved for your application against your LaunchDarkly dashboard. ~ Review, Copy and reset SDK credentials for an environment

2. Are you seeing WARN: Error in stream connection (will retry): Get "https://stream.launchdarkly.com/all": Forbidden or a similar Forbidden error?

   • Yes: this means the LaunchDarkly service endpoints are not yet allowlisted in your organization's network firewall, the application server Content Security Policy, or both. You may need to inform the team that manages the network or application allowlist that your team or department uses LaunchDarkly and explain how it will be used in your application or server. ~ Review Common misconceptions about LaunchDarkly architecture

3. Does your traffic go through a reverse proxy, also known as a web server that proxies HTTP requests?

   • Yes: Check if the reverse proxy allows long-lived connections.
   • No: Are you seeing errors like Stream received invalid data in PUT or Malformed JSON data in event stream?
     • Yes: confirm whether this reverse proxy or web server is adding or modifying HTTP headers. Unknown HTTP headers added in the request by a reverse proxy are a common cause of these errors.

4. Do you have different firewalls or web proxies based on the app/process environment (e.g., one firewall for dev and a different firewall for production)?

   • Yes: Compare the configurations of a working environment against those of the non-working environments to identify differences. If the working environment does not have connectivity problems, this indicates that the problem is due to different firewalls that are not configured in the same way. 

5. Are there SSL errors or mentions of a certificate issue in the error logs?

   • Yes: Check with the team that manages the application hosting to verify that you use a managed trust store or certificate authority. You may need to update the custom trusted certificate store to recognize LaunchDarkly. For Java, review this knowledge base article.

6. Is the SDK connecting to a Relay Proxy?

   • No: Proceed to Step 7.
   • Yes: Is it receiving a 503 response?
     • Yes: This means the Relay Proxy isn't in a ready state yet. In this case, all LaunchDarkly SDKs connecting through the Relay Proxy instance will retry after a backoff delay:
7. Is the issue explained by our Common misconceptions about LaunchDarkly architecture help article?
   • No: Contact Technical Support and provide additional context with the troubleshooting steps you have taken.

Connection Interruption Troubleshooting

1. Is there a load balancer between the SDK and Relay Proxy?

   • Yes: Ensure traffic is rebalanced after changes to the Relay Proxy instance.

2. Does traffic go through a reverse proxy?

   • Yes: Verify that connections are not prematurely terminated.

3. Are there frequent timeouts or similar errors? Example: 

[LaunchDarkly] Received I/O error (Read timeout, received no data in Xms, assuming connection is dead) for streaming request - will retry

[LaunchDarkly] Will retry stream connection in 1000 milliseconds

• • Yes: your network is closing the streaming connection between the SDK/Relay Proxy instance and LaunchDarkly. ~ Review, Frequent timeouts of streaming connection

Resources

• Getting Started with SDKs
• Logging Configuration
• LaunchDarkly Contributor's Guide
• LaunchDarkly Domain List
• LaunchDarkly Public IP List