Affected: Server-side SDKs, Relay Proxy
Symptoms
The following symptoms occur when initializing or occur intermittently while running the Relay Proxy/SDK when connected to LaunchDarkly:
- SDK serving stale variations
- Frequent messages indicating network issues
Cause
Your application may be creating too many SDK instances, leading to resource exhaustion and an inability to open new connections to LaunchDarkly.
If timeouts occur on a host but are not reproducible using sample applications locally or through automated testing, a bottleneck may be causing the requests to time out.
Possible bottleneck causes:
- Over-instantiating the SDK: Too many SDK instances running at the same time.
- Too many streaming connections per host: Containerized environments may create excessive connections.
- Internal network congestion: High numbers of streaming connections affecting network stability.
Solution
Before setting up a Relay Proxy, confirm whether you are over-instantiating the SDK. The Relay Proxy can mask SDK overuse symptoms, leading to larger issues in the future.
Follow these troubleshooting steps in order:
1. Too many SDK instances due to over-instantiation
Excessive SDK instantiation can be difficult to detect, especially if instances are accidentally created without proper reference tracking.
Leaked SDK instances continue attempting to maintain streaming connections, gradually consuming system resources. This can lead to failures in maintaining streaming connections and cause the Too many open files error. Too many open files error occurs when a process exceeds the OS limit for open file handles and socket connections.
Default limits:
-
- MacOS: 256
- Windows: 512
- Linux: 1024
To address these concerns, make sure you implement the SDK as a singleton.
2. Too many streaming connections per host due to containerization
Containerized environments may create excessive streaming connections, leading to bottlenecks and timeouts.
Use a node/host-level Relay Proxy to manage connections. The Relay Proxy maintains a single streaming connection to LaunchDarkly while serving SDKs on the same node. This reduces the number of direct SDK connections to LaunchDarkly, preventing network congestion.
This could also result in the Too many open files error. See the explanation above.
Example:
Set up a container running the Relay Proxy and connect all SDKs on that node to the container. Network requests between containers require less overhead than direct connections to LaunchDarkly.
3. Too many streaming connections causing internal network congestion
If a node/host-level Relay Proxy has been implemented and timeouts persist, the issue may stem from network-level bottlenecks.
Deploy a network-level Relay Proxy that connects all nodes/hosts. This reduces outgoing traffic to a single streaming connection, minimizing internal network congestion.
Resources