Load Balancing
| Client side Load Balancing is an active work-in-progress, it’s used in production, but expect improvements in the future. |
One of the core abstractions in ServiceTalk is its Client side LoadBalancer underpinning the various protocol Clients (eg. HTTP/1.1, HTTP/2, gRPC, etc.).
For scenarios where a client communicates with multiple homogeneous [1] server instances, managing multiple clients for each service instance is complex. Client side load balancing makes this easy for the users by transparently communicating with multiple servers. It also provides opportunities to spread the load across all the servers, minimize latency and reduce error rates thus providing a resilient and performant system.
Architecture
ServiceTalk’s protocol-specific Clients and LoadBalancer are layered as follows:
+-------------------+
| ConnectionFactory |
+-------------------+ +--------------+ +----------------------+ +--------+
| /--->| Connection 1 |<--->| HTTP Decoder/Encoder |<--->| Socket |
V | +--------------+ +----------------------+ +--------+
+--------+ request +--------------+ |
| HTTP |--------->| Client-Side | | +--------------+ +----------------------+ +--------+
| Client | | LoadBalancer |<---+--->| Connection 2 |<--->| HTTP Decoder/Encoder |<--->| Socket |
| |<---------| | | +--------------+ +----------------------+ +--------+
+--------+ response +--------------+ |
^ | +--------------+ +----------------------+ +--------+
| \--->| Connection x |<--->| HTTP Decoder/Encoder |<--->| Socket |
+-------------------+ +--------------+ +----------------------+ +--------+
| ServiceDiscoverer |
+-------------------+
The load balancer provides similar functionality to a traditional connection pool, but it is consulted on each request and can be influenced by protocol feedback. This enables building a rich set of features on top such as: intelligent capacity management, smart connection and host selection, circuit breaking, etc. The Connection abstraction which is the basis for all protocol-specific Connections allows the LoadBalancer implementations to be reusable across all the protocols.
The LoadBalancer is able to combine availability information from ServiceDiscoverer and protocol metrics from the ConnectionFactory (e.g. latency, …) in order to pick a more optimal Connection for each request.
Implementation
As mentioned earlier the Client-Side LoadBalancer abstraction allows for various protocol-independent LoadBalancing algorithms to be implemented. However, the built-in default Load Balancer is a highly featured implementation that we strongly recommend and creating custom load balancing solutions should be done as a last resort.
Default Load Balancer
The ServiceTalk default Load Balancer is the recommended load balancer. It is a modularization of the RoundRobinLoadBalancer intended to support features necessary for proxyless service-to-service communication. It supports multiple selection algorithms including Power of Two Choices (P2C) and Round-Robin as well as layer-7 outlier detection mechanisms including consecutive failure and outlier detection.
See the default Load Balancer documentation for more detail.