Load Balancing
| Client side Load Balancing is an active work-in-progress, it’s used in production, but expect improvements in the future. |
One of the core abstractions in ServiceTalk is its Client side LoadBalancer underpinning the various protocol Clients (eg. HTTP/1.1, HTTP/2, gRPC, etc.).
For scenarios where a client communicates with multiple homogeneous [1] server instances, managing multiple clients for each service instance is complex. Client side load balancing makes this easy for the users by transparently communicating with multiple servers. It also provides opportunities to spread the load across all the servers, minimize latency and reduce error rates thus providing a resilient and performant system.
Architecture
ServiceTalk’s protocol-specific Clients and LoadBalancer are layered as follows:
+-------------------+
| ConnectionFactory |
+-------------------+ +--------------+ +----------------------+ +--------+
| /--->| Connection 1 |<--->| HTTP Decoder/Encoder |<--->| Socket |
V | +--------------+ +----------------------+ +--------+
+--------+ request +--------------+ |
| HTTP |--------->| Client-Side | | +--------------+ +----------------------+ +--------+
| Client | | LoadBalancer |<---+--->| Connection 2 |<--->| HTTP Decoder/Encoder |<--->| Socket |
| |<---------| | | +--------------+ +----------------------+ +--------+
+--------+ response +--------------+ |
^ | +--------------+ +----------------------+ +--------+
| \--->| Connection x |<--->| HTTP Decoder/Encoder |<--->| Socket |
+-------------------+ +--------------+ +----------------------+ +--------+
| ServiceDiscoverer |
+-------------------+
The load balancer provides similar functionality to a traditional connection pool, but it is consulted on each request and can be influenced by protocol feedback. This enables building a rich set of features on top such as: intelligent capacity management, smart connection and host selection, circuit breaking, etc. The Connection abstraction which is the basis for all protocol-specific Connections allows the LoadBalancer implementations to be reusable across all the protocols.
The LoadBalancer is able to combine availability information from ServiceDiscoverer and protocol metrics from the ConnectionFactory (e.g. latency, …) in order to pick a more optimal Connection for each request.
Implementations
As mentioned earlier the Client-Side LoadBalancer abstraction allows for various protocol-independent LoadBalancing algorithms to be implemented. This section will discuss the various implementations offered in ServiceTalk by highlighting their characteristics.
Round Robin
RoundRobinLoadBalancer is a common and simple LoadBalancer implementation that is currently the default when creating Clients. Its main goal is to spread the load evenly between all known resolved addresses as provided by the Service Discovery mechanism.
The implementation in ServiceTalk consists of a set of available addresses (typically the servers to connect to) and for
each address it has a set of open Connections. Whenever a new request is made the LoadBalancer will pick the next
address from its list of addresses and picks one of its open Connections until it finds an available Connection.
When all Connections are in use, it’ll try to open a new Connection to that same address. It works in tandem with
ServiceDiscoverer, when new addresses are added or addresses are removed it’ll update its active addresses set for
future Connection selection. This approach ensures that every address will receive an equal amount of requests on
average across all Clients.
| This approach favors lower selection time over lowering latency and error rates. |